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Preface 


The reason for writing this eHandBook is to provide a short easy to read 
introduction to regulatory compliance, licensing measurement and 
monitoring systems for licensing researchers and administrators, and for 
regulatory scientists and policymakers. It is not intended to be a 
comprehensive, in-depth analysis of regulatory compliance and licensing 
measurement. This eHandBook provides only a basic introduction to 
licensing measurement. For those readers who are interested in doing a 
deeper dive into licensing measurement, I recommend NARA's - National 
Association for Regulatory Administration's Licensing Curriculum and their 
online courses offered through the University of Southern Maine. 


This eHandBook will provide the basics to get the interested learner pointed 
in the direction of learning more about the topic. As one will see regulatory 
compliance and licensing measurement has its challenges and idiosyncrasies 
which will need to be addressed by researchers and scientists. When I started 
this journey 50 years ago, I was somewhat taken aback by the different data 
distributions I encountered in regulatory science. Back then, regulatory 
science was not well formulated and program monitoring related to licensing 
was more qualitative (case notes) rather than quantitative. It was also very 
uniform in its application. But I got really interested in public policy and 
macro-systems which seemed to have more and more impact on children 
and their families. This was the beginning of governmental rule 
promulgation, and it was an exciting time to be on the cusp of this new 
research area. 


I learned very quickly that I had to make several adjustments to the statistical 
methods I learned in graduate school to be able to analyze licensing data and 
measure regulatory compliance. Several of the theories and methodologies 
were controversial when I proposed them because they went counter to the 
prevailing paradigms at that point in the 1960-70's. However, over time with 
many replications and validations, the new conceptual framework was 
accepted in the licensing and the regulatory science research literatures. 


There are many people to thank over the years and obviously this has been 
a group effort in applying regulatory science to early care and education and 
then expanding it to human services and hopefully beyond. What I have 
found in my most recent readings is that regulatory science is being applied 
in many different content silos from the FDA, to economics, to banking, and 
of course within the human services, particularly adult and child residential 
services. What appears to be lacking is a unifying theory that goes across 
these disparate content areas. That is why I think the introduction of the 
Regulatory Compliance Theory of Diminishing Returns is such an important 
contribution when we think about licensing/regulatory measurement and 
monitoring systems. The theory has become the foundation for the 
development of the methodologies and metrics presented in this eHandBook 
such as key indicators, risk assessment, differential monitoring, instrument- 
based program monitoring, integrative quality monitoring, skewed data 
distributions, nominal and ordinal measurement scaling, how best to deal 
with false negatives in decision making, and the balancing act between 
regulatory compliance & quality programming. 


As I said earlier this eHandBook needs to be read along with the published 
materials on the Research Institute for Key Indicators: RIKI 
(https://RIKInstitute.com) and the National Association for Regulatory 
Administration: (https://www.naralicensing.org/key-indicators) websites. It 


is not intended as a standalone text for licensing measurement. 


So, let’s start this intriguing journey together into the world of regulatory 
compliance in order to understand the idiosyncracies and nuances of this 


emerging field within regulatory science. 


Rick Fiene, PhD, Research Psychologist & Regulatory Scientist 

Research Institute for Key Indicators, Edna Bennett Pierce Prevention Research 
Center, Penn State University, & National Association for Regulatory 
Administration 

July 2024 


Chapter 1 


Introduction to Regulatory Compliance, Licensing 
Measurement and Monitoring Systems 


This first chapter provides the learner with an introduction and 
overview to regulatory compliance, licensing measurement and 
systems. The ehandbook is sponsored by NARA — National 
Association for Regulatory Administration. NARA is the 
prominent international organization dealing with human 
services licensing and regulatory administration. This 
ehandbook is part of the NARA Licensing Curriculum which 
you can find out more about by visiting NARA's website 
(https://www.naralicensing.org/nara-licensing-curriculum). 

NARA also offers a course by the same name and this book is 
the eTextBook for that course; it is intended to be used in 
conjunction with the NARA Licensing Measurement course. 


The NARA course will provide the learner with the major tenets 
of licensing measurement. The learner will discover as they go 
through this book that measurement in licensing is very 
different than other measurement systems found in many of the 
various social and human services and sciences. It has some 
very unique and idiosyncratic aspects which will provide us 
with increasing challenges in coming up with specific metrics 
in determining regulatory compliance. 


The field of regulatory science is a very young field. Although 

regulations have been kicking around for well over 100 years, 

the science behind regulations is probably a quarter of this time. 

So, there is not a great deal of empirical evidence to draw upon 
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which is discouraging but it is very encouraging and exciting at 
the same time because so much needs to be accomplished in 
establishing regulatory science theory. 


A great deal has been written in the past 20 years about 
regulatory science but there has not been a book written about 
measurement and regulatory compliance. It is hoped that this 
book will begin that discussion. It is also hoped that data driven 
via regulatory science will begin to inform regulatory 
administration and policy more clearly as we move forward. 


This specific chapter will provide the conceptual framework 
and overview to regulatory compliance, licensing measurement 
and systems of regulatory compliance. It will provide the 
parameters of the book's organization and what will be covered 
throughout. 


The other chapters to be covered in this book are the following: 


1. Overview/Introduction 

2. Conceptual/Theoretical Framework 

3. Principles of Instrument Design 

4. Regulatory Compliance and Program Quality 

5. Coordinated, Differential Monitoring & Integrated Monitoring 
6. What Research Tells Us; What Research Doesn’t Tell Us 

7. Future Directions 


The book is organized into the above 7 chapters and appendices. 
The book is short and provides the basics to licensing 
measurement and systems. It is a quick read for regulatory 
scientists and regulatory policy makers as well as licensing 
administrators. It can be read as a standalone text although it 
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was intended as the textbook for the NARA Licensing 
Measurement course, and it is recommended to be used with 
that course. 


The NARA Licensing Measurement course is approximately 15 
hours in length and is organized into anywhere from 5 - 10 
classes. It is equivalent to a one-credit course offered at most 
institutions of higher education. Each class is organized into 
the following: an overview to what will be covered in the 
specific class followed by annotated PowerPoint slides, 
followed by a series of readings to support the specific 
lecture/PowerPoint slides (I will be referring to these various 
resources throughout this text). For the learner who wants to 
get a thorough grounding in licensing measurement and its 
accompanying program monitoring systems, I highly 
recommend them taking the course. 


This book and the course are self-paced and are geared to the 
individual learner. It is totally self-contained meaning that all 
the necessary content is contained with the ten classes. If a 
learner just wants to get an overview of what licensing 
measurement is all about, then reading this short ebook will be 
a great start. You can always check out any of the publications 
that are available on the RIKI _ Institute website 
(https://rikinstitute.com/publications/). However, if a learner 
does have a specific question related to this textbook or if they 
are interested in taking the course and would like to get in touch 
with Dr Fiene, here is his contact information to reach out 
(email address is the best way to contact Dr Fiene): 


Dr Richard Fiene, Research Psychologist & Emeritus Professor 
of Psychology & Human Development 
Research Institute for Key Indicators (RIKI) & Penn State 
University 
RFiene @ RIKInstitute.com 
Senior Research Consultant/Senior Regulatory Scientist 
National Association for Regulatory Administration 


RFiene@ NARALicensing.org 


Also, this ehandbook and the course will draw heavily from 
both the NARA and RIKI websites where many of the 
publications and research reside. Please feel free to go to 
https://RIKInstitute.com/blog/ to download any additional blog 
posts that may be of interest to you. As I said, all the research 
is in the public domain and follows an open science sharing 
arrangement. 


The links for additional publications (see the references listed 
at the end of this book) for the NARA course are listed 
throughout this book or within the course format in the 
handouts section on the NARA website which you can 
download in their entirety or do it chapter by chapter. All 
course materials will be provided in either the lectures section 
or the handouts section of the class. 


I thought it would be helpful to provide a bit of my background 
which will help the reader to put in context the content of this 
text and the course. This textbook is written by a research 
psychologist who has spent his career in improving childcare 
quality through an early childhood program quality indicator 
model of training, technical assistance, quality rating and 
improvement systems, professional development, mentoring, 
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licensing, risk assessment, differential program monitoring, key 
indicators, and accreditation. While content wise, I spent my 
career in early care and education, I evolved into a regulatory 
scientist because of the various positions I held within 
governmental service and interest area that focused on public 
policy, macro systems, and licensing rules. 


Here are some additional commentaries taken from my NIH Bio 
Sketch that might help to fill in some details related to my 
background: 


Dr Fiene is an emeritus professor of human development & 
psychology (Penn State University) where he was department 
head and founding director of the Capital Area Early 
Childhood Research and Training Institute. He is presently 
President & Senior Research Psychologist/Regulatory Scientist 
for the Research Institute for Key Indicators. 


Dr Fiene is regarded as a_ leading international 
researcher/scholar on human services licensing measurement 
and differential monitoring systems. His regulatory compliance 
law of diminishing returns has altered human services 
regulatory science and licensing measurement dramatically in 
thinking about how best to monitor and assess licensing rules 
and regulations through targeted and abbreviated inspections. 


His research has led to the following developments: 

identification of herding behavior of two-year old's, national 

early care and education quality indicators, mathematical 

model for determining adult child ratio compliance, solution to 

the trilemma (quality, affordability, and accessibility) in child 

care delivery services, Stepping Stones to Caring for Our 
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Children, online coaching as a targeted and individualized 
learning platform, validation framework for early childhood 
licensing systems and quality rating & improvement systems, 
an Early Childhood Program Quality Improvement & Indicator 
Model, Caring for Our Children Basics, and has led to the 
development of statistical techniques for dealing with highly 
skewed, non-parametric data distributions in human services 
licensing systems. 


Organization of the eHandBook 


In the following chapters, the reader will find how licensing 
measurement is very unique when it comes to regulatory 
science. By knowing these unique characteristics, it will be 
easier to administer and monitor programs governed by 
licensing rules. This can be of benefit to those who administer 
licensing agencies and to those who are asked to conduct 
research on regulatory policies and compliance. 


In Chapter 1, this chapter, provides the basic introduction and 
overview to the ehandbook as well as a short history of 
licensing measurement and a timeline for early care and 
education standards focusing on Caring for Our Children: 
Basics (CFOCB). CFOCB is a key document and set of 
standards providing a core set of rules that can govern all early 
care and education programs which is the goal of any 
monitoring system. 


In Chapter 2, the reader will be introduced to the theoretical and 

conceptual foundations to regulatory compliance and licensing 

measurement. The regulatory compliance theory of 

diminishing returns will be introduced which has had a 
12 


tremendous impact on human services regulatory science and 
administration. Paradigm alternatives will be suggested that 
have guided regulatory science over the past several decades. 


In Chapter 3, the principles of instrument design will be 
addressed. Obviously without a reliable and valid system of 
measurement it will be simply garbage in and garbage out. This 
is an area when it comes to instrument design that gets the short 
end of the stick many times. The level of measurement will be 
addressed and its impact on the types of statistics selected. 
Also, how best to design data bases will be addressed. 


In Chapter 4, regulatory compliance and program quality will 
be discussed presenting it on a dichotomous ten-point polemic 
for the regulatory science field to consider. This is a very 
important chapter building off the theory of regulatory 
compliance of diminishing returns introduced in chapter 2. 


In Chapter 5, the essence of program monitoring systems is 
introduced along with differential monitoring and its two major 
methodologies of key indicators and risk assessment. This 
chapter gets us thinking about what a licensing measurement 
will look like administratively. 


Chapter 6 deals with the research literature what we know and 
what still needs to be addressed, the gaps in our regulatory 
science knowledge base. Examples are provided of success 
stories across the USA and internationally. 


The last chapter, Chapter 7, provides us with where do we go 
from here with regulatory compliance, licensing measurement 
and monitoring systems. What are the next steps? 
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There are a series of technical research notes that form an 
appendix which help to clarify the body of the text and add 
some details. This is followed by a comprehensive reference 
listing and then several figures, charts, graphs, and displays 
which depict various concepts presented in the text. And lastly, 
there is an example of a validation study report from 
Saskatchewan which provides the reader with how all this 
would look when the contents of this ehandbook are utilized 
and played out in a research study. 


A _ Brief History of Licensing Measurement, Monitoring 
Systems and Regulatory Compliance 


The history of licensing measurement and_ regulatory 
compliance has a rather long lineage but is still in its infancy in 
terms of development. In the early stages most licensing visits 
and inspection results were recorded via anecdotal records/case 
records with the licensing staff recording their results in more 
social work note taking. It was a qualitative type of 
measurement with very little quantitative measurement 
occurring except for basic demographics, number of clients, 
number of caregiving staff, etc... This qualitative approach 
worked very well when there were not many programs to be 
monitored and there were sufficient licensing staff to do the 
monitoring and conduct the inspections. 


This all started to change in the 1980’s when Instrument Based 
Program Monitoring (IPM) was introduced and started to be 
adopted by state licensing agencies throughout the United 
States. Just as a footnote, this brief history is pertinent to the 
USA and does not include other countries although the 
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Canadian Provinces have followed a similar route as the USA. 
The reason for the introduction of an IPM approach was the 
tremendous increase in early care and education programs in 
the 1960’s and 1970’s. It was difficult for licensing staff to keep 
up with the increased number of programs in their monitoring 
efforts. There needed to be a more effective and efficient 
methodology to be employed to deal with these increases. 


A very influential paper was written in 1985 and published in 
Child Care Quarterly which introduced IPM along with 
Licensing Key Indicators, Risk Assessment (Weighting), and 
Differential Monitoring (Abbreviated Inspections). This paper 
outlined the various methodologies and their use by a 
consortium of states to test the viability of this new approach to 
licensing measurement, regulatory compliance, and program 
monitoring. Also, the terminology has changed over the 
decades. Back in 1985 weighting was used rather than risk, 
abbreviated inspections were used rather than differential 
monitoring, targeted monitoring, or inferential monitoring. All 
these terms can be used interchangeably as they have been over 
the years, but the first introduction of them back in 1985 utilized 
weighting and abbreviated inspections. 


In the early 1990’s the risk assessment methodology was used 
to develop Stepping Stones to Caring for Our Children, the 
comprehensive national health and safety standards for early 
care and education (ECE) programs in the USA. This was a 
major development in attempting to develop national voluntary 
standards for child care in the USA. 


It was during this time that two other very significant 
discoveries occurred related to licensing data distributions: 1) 
Licensing data are extremely skewed and do not follow a 
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normal curve distribution. This fact has a significant impact on 
the statistics that can be used with the data distributions and 
how data analyses are performed. For example, data 
dichotomization is warranted with licensing data; 2) Regulatory 
compliance data are not linear when compared to program 
quality measures but are more plateaued at the substantial and 
full regulatory compliance levels. The data appear to follow the 
Law of Diminishing Returns as compliance moves from 
substantial to full (100%) regulatory compliance. This finding 
has been replicated in several studies and has been controversial 
because it has led to the issuing of licenses to programs with 
less than full compliance with all rules/regulations/standards. 
These two discoveries have been very influential in tracking 
developments in licensing measurement since their discoveries. 


In the new century as states began to adopt the various 
methodologies it became necessary to have a standardized 
approach to designing and implementing them. The National 
Association for Regulatory Administration (NARA) took up 
this role and in 2000 produced a chapter on Licensing 
Measurement and Systems which helped to guide 
states/provinces in the valid and reliable means for designing 
and implementing these methodologies. In 2002 a very 
important study was conducted by the Assistant Secretary’s 
Office for Planning and Evaluation (ASPE) in which they 
published the Thirteen Indicators of Quality Health and Safety 
and a Parent’s Guide to go along with the research. This 
publication further helped states as they revised their licensing 
and program monitoring systems for doing inspections of early 
care and education facilities based upon the specific indicators 
identified in this publication. Both publications have been 
distributed widely throughout the licensing world. 
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During the first decade of the new century, Stepping Stones for 
Caring for Our Children went through a second edition. This 
publication and the ASPE publications were very useful to 
states as they prepared their Child Care Development Fund 
(CCDF) plans based upon Child Care Development Block 
Grant (CCDBG) funding. 


From 2010 to the present, there have been many major events 
that have helped to shape licensing measurements for the future. 
Caring for Our Children Basics (CFOCB) was published and 
immediately became the default voluntary early care and 
education standards for the ECE field. The CFOCB is a 
combination of the risk assessment and key indicator 
methodologies. Three major publications by the following 
Federal agencies: HHS/ACF/USDA: Department of Health and 
Human — Services/Administration for Children and 
Families/United States Department of Agriculture, OCC: 
Office of Child Care, and ASPE: Assistant Secretary’s Office 
for Planning and Evaluation dealing with licensing and program 
monitoring strategies were published. These publications will 
guide the field of licensing measurement for years to come. The 
Office of Head Start developed and implemented their own 
Head Start Key Indicator (HSKI) methodology. And in 2016, 
CCDBG was reauthorized and differential monitoring was 
included in the legislation being recommended as an approach 
for states to consider. 


Most recently, the Office of Head Start is revising their 

monitoring system that provides a balance between compliance 

and performance. This system revision will go a long way to 

enhancing the balance between regulatory compliance and 

program quality. Also, there has been experimentation with an 

Early Childhood Program Quality Indicator instrument 
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combining licensing and quality indicators into a single tool. 
These two developments help with breaking down the silo 
approach to measurement where licensing and quality 
initiatives are administered through separate and distinct 
approaches such as licensing versus professional development 
systems versus quality rating and improvement systems. A 
paradigm shift in which an Early Childhood Program Quality 
Improvement and Indicator Model is proposed. The paradigm 
shift should help to make licensing measurement more 
integrated with other quality initiatives. 


The licensing field continues to make refinements to its 
measurement strategies in building a national/international 
regulatory compliance data base. More and more is being 
learned about the nuances and idiosyncrasies of licensing data, 
such as moving from a nominal to an ordinal driven data 
system. For example, NARA and the Research Institute of Key 
Indicators (RIKI) have entered into an exclusive agreement for 
the future development of licensing measurement strategies via 
differential monitoring, key indicators for licensing and 
program quality, and risk assessment approaches. Several 
validation studies have been completed in testing whether the 
various methodologies work as intended. A significant Office 
of Program Research and Evaluation (OPRE) Research Brief 
which developed a framework for conducting validation studies 
for quality rating and improvement systems has been adapted 
to be used in licensing measurement. 


For additional updates to licensing measurement, please check 

out and follow these RIKINotes Blog posts. There are and will 

be many examples of licensing measurement enhancements. 

Also, although much of the research on licensing measurement 

has been completed in the ECE field, the methodologies, 
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models, systems, and approaches can be utilized in any human 
service arena, such as child residential or adult residential 
services. Also, NARA’s chapter in their Licensing Curriculum 
has been developed into a full-blown course, please go to the 
following web page for additional information 
(https://www.naralicensing.org/key-indicators). 


A Timeline of ECE Standards and Program Monitoring in 
the United States: Caring for Our Children Basics—probably 
the best example of a health and safety standards tool 


It all started in and around 1965 when the Federal government 
got into early care and education (ECE) in earnest with Head 
Start and federally funded day care for low-income families. It 
started off slowly but began to pick up momentum with exciting 
studies and research applying principles from developmental 
psychology to policy making. Researchers and policy makers 
wanted to make sure that these new programs were not 
detrimental to young children since our frame of reference were 
children being raised in orphanages and the ultimate outcome 
for children was not positive. Would ECE have the same 
impact? 


Issues around quality, appropriateness of standards, and 
demonstration programs became the focal point of federal 
research funding. The focal point of this essay is on the 
appropriateness of the ECE standards and the resulting 
monitoring systems that were to become key to the federal 
involvement in early care and education. This essay will be 
organized by the following 50 years neatly broken out by each 
decade to get us from this beginning in 1965 until the 
publication of Caring for Our Children Basics in 2015 by the 
federal government, the Administration for Children and 
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Families, U.S. Department of Health and Human Services. A 
look at the 2020 decade with a future note is also appended to 
this essay. 


1970s 


During the 1970’s, the federal government became concerned 
about what were to be the standards for this new national 
program related to federally funded ECE for low-income 
families and their children. Head Start was a separate entity and 
we will revisit Head Start later but our focus for now is on the 
federally funded programs which became known back then 
generically as day care. This nomenclature changed to child 
care and to finally early care and education (ECE) during this 
50-year history. The initial standards for day care were the 
Federal Interagency Day Care Requirements (FIDCR). A very 
large appropriateness research study led by Abt Associates to 
determine what were the most salient standards and their 
intended impact on children while in day care was conducted 
during this decade. These standards were to be federally 
mandated requirements for any program receiving federal 
funding. This is where group size and adult-child ratios 
standards became such important safeguards and surrogates for 
children’s health and safety in day care programs. 


It also became of interest for the federal government to design 
the monitoring system that would determine compliance with 
the FIDCRs. But it became clear to the original designers of 
this new system that the monitoring of the FIDCR was going to 
be difficult to do across the full USA. So, the question became, 
is there a way to monitor the standards in the most effective and 
efficient manner? This question and the future of the FIDCR 
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were to be altered and put on hold once we moved into the next 
decade. 


1980s 


A change in federal administration and a resulting change in 
philosophy related to the federal role in America altered many 
things and one of them was the relationship of the federal 
government and the states. Rather than the federal government 
mandating day care requirements, the focus changed with the 
locus of control moving from the federal level to the state level 
via block grant funding with very few federal requirements. 
This meant a moratorium to FIDCR and its ultimate demise. 
The federal government was not going to be in the business of 
providing day care, this was going to be the jurisdiction of the 
states. Head Start did become the exception to this rule with its 
own standards and monitoring system. 


The focus of federal funding switched from the national to the 
state level in determining compliance with each state’s 
respective child care licensing rules and not with an overarching 
FIDCR. There was still interest in making these state 
monitoring systems as effective and efficient but there was no 
interest in the federal government determining what these 
requirements would be. Two monitoring approaches grew out 
of this need for effectiveness and efficiency: risk assessment 
and key indicators. | These two approaches were originally 
designed and implemented as part of a federally funded project 
called the Children’s Services Monitoring Transfer Consortium 
in which a group of five states: New York, Michigan, 
Pennsylvania, West Virginia, and California teamed up to 
explore their most effective and efficient monitoring systems 
and begin transferring these systems to one another and beyond. 
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These two monitoring approaches were tested in the above 
respective states and it was determined that their impact had a 
positive effect on the children who were in those day care 
centers. This was a major finding, similar to the FIDCR 
appropriateness study, in which these approaches provided 
safeguards related to the health and safety of children while in 
day care. 


1990s 


By the 1990s, it became clear that the federal government had 
pretty much drawn back from any leadership role in having 
mandated federal requirements when it came to health and 
safety in child care. It was left to national ECE advocates who 
were positioned within the federal government (Administration 
for Children and Families; Maternal and Child Health Bureau) 
as well as throughout the USA with national and state agencies 
and organizations (American Academy of Pediatrics; American 
Public Health Association, National Resource Center for Health 
and Safety in Child Care) that saw a need for child care health 
and safety recommendations at least. If we could not have 
requirements, we could at least have recommendations and 
provide guidance to child care programs throughout the USA. 


This led to the first edition of Caring for Our Children which 
was a comprehensive set of childcare health and safety 
standards. It was a major game changer for the ECE field 
because now there was a universal set of standards based upon 
the latest research literature for states to use as they considered 
revising and updating their respective state licensing childcare 
rules. 


But there was a problem. Caring for Our Children was a 
comprehensive set of health and safety standards which was 
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their strength but at the same time it was their weakness. They 
were so comprehensive (well over 500 well researched 
standards) that they were intimidating, and it was difficult to 
determine where to begin for the states. 


Several researchers remembered the two approaches to 
monitoring designed in the previous decade and wondered if 
they could be helpful in focusing or targeting which of the 
standards were the most critical/salient standards. The risk 
assessment approach to monitoring appeared to have the most 
immediate applicability and Stepping Stones to Caring for Our 
Children was born. This document clearly articulated which of 
the 500+ Caring for Our Children standards placed children at 
greatest risk for mortality or morbidity by not being in 
compliance with the respective standard. Since the early 1990s, 
Caring for Our Children and Stepping Stones to Caring for Our 
Children have gone through three editions and have become 
very important resources to state licensing agencies as they 
revise, update and improve their ECE rules. 


2000s 


In this decade several federal and national organizations began 
to use Caring for Our Children standards in innovative ways to 
measure how well ECE looked at a national level. The 
Assistant Secretary’s Office for Planning and Evaluation in the 
U.S. Department of Health and Human Services published the 
Thirteen Indicators of Quality Child Care based upon a core set 
of predictor standards from Caring for Our Children. These 
were standards that predicted overall compliance with all the 
standards and were seen as an efficient monitoring system. 
NACCRRA (National Association for Child Care Resource and 
Referral Agencies) began publishing a national report card on 
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how well states met specific standards and monitoring protocols 
based upon similar predictor standards from Caring for Our 
Children. 


These efforts helped states to make significant changes in their 
ECE rules in their respective states and in a very voluntary way 
suggested a means for national standards for the ECE field 
although we would need to wait until the next decade in order 
to see such a published document of national ECE health and 
safety standards for early care and education: Caring for Our 
Children Basics. It would not be until 2015 when Caring for 
Our Children Basics was published by the Administration for 
Children and Families, U.S. Department of Health and Human 
Services. 


2010s 


By the 2010s, ECE had grown into a very large but unwieldly 
assortment of programs with varying levels of quality. Again 
because of major federal funding, the Child Care Development 
Block Grant, along with changes and enhancements in 
professional development, accreditation systems, quality rating 
and improvement systems, the ECE landscape had become 
more complex and less easy to navigate. And rather than 
coming together it was clearly more fragmented than ever. 


We had very minimal requirements for the federal funding and 
most of these requirements were geared to the state agency 
using the state’s respective licensing rules as the threshold for 
standards. This approach worked well with states with 
excellent licensing rules, but it wasn’t working as well with 
states who did not have equally excellent licensing rules. We 
still did not have a core set of standards for ECE programs. 
Enter Caring for Our Children Basics which took the best 
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aspects of the above two monitoring approaches, risk 
assessment and key indicators and molded it into this new 
document. This work was led by the federal government’s 
Administration for Children and Families, U.S. Department of 
Health and Human Services and although the standards are still 
recommendations and guidance, it is our best attempt at having 
national standards for early care and education. It is an attempt 
to provide guidance to the full ECE field, child care, Head Start, 
preschool, and center based as well as home-based care. It 
would be nice to have Caring for Our Children Basics as the 
health and safety foundation for early care and education 
throughout the USA. I don’t see this happening in my lifetime. 


2020s: Looking to the Future 


As a footnote to this essay, the new decade has been dealt with 
a major curve ball with COVID19 rearing its ugly head and 
ECE has been impacted greatly because of this pandemic. As 
of this writing we are nowhere closer to a solution to getting 
ECE programs back on line. If anything, the pandemic really 
demonstrated the fragility of the ECE system we have built over 
the past 50 years and it clearly has not done very well. My hope 
is that we can learn from the past 50 years and not continue 
another 50 years along the same route; although I am guessing 
that many ECE advocates would be glad to have what we had 
before the pandemic because what we have right non- 
sustainable. We know a lot more today than what we knew 
back in 1965 when we were worried about would day care hurt 
children’s development. We know today that quality ECE 
benefits children but unfortunately, we are no closer to attaining 
this today than we were 50 years ago. 
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Two programs that have been very successful in avoiding these 
pitfalls are Head Start and the national Military Child Care 
program. Both programs are exemplary examples of quality 
early care and education being provided with separate funding 
streams and_ standards. Interesting enough when _ the 
Administration for Children and Families published Caring for 
Our Children Basics, both these programs were part of the 
reach of the published standards. As we re-invent and re- 
structure ECE we should be looking to both these very 
successful programs for guidance. 


The above history and timeline have been drawn from the early 
care and education field but very honestly that is where the 
majority of the research and development has been over the past 
50 years related to regulatory compliance. There have been 
some excursions into other human services but they don’t have 
the details as with early care and education. 
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Chapter 2 
Conceptual and _ Theoretical Underpinnings, 
Program Monitoring Paradigms 


This second chapter provides the learner with the key 
conceptual and theoretical foundations related to licensing 
measurement. As you have seen from the first chapter and will 
continue to see throughout this book, licensing measurement 
does have some idiosyncrasies which are not present in other 
data distributions. 


Well, the same thing can be said when it comes to the 
conceptual and theoretical underpinnings. One of the first 
limitations that will be noted is the regulatory compliance 
theory of diminishing returns which has _ tremendous 
implications when implementing and enforcing rules. It had 
always been assumed that full 100% regulatory compliance 
with rules was what made a high-quality program. However, in 
the late 1970's and into the early 1980's, it became clear that this 
was not the case. When this hypothesis was tested it became 
clear that moving from low regulatory compliance to 
substantial regulatory compliance did demonstrate that program 
quality differed significantly in the substantial regulatory 
compliant programs being of a higher quality than those of 
lower regulatory compliance. However, when one moved from 
the substantial regulatory compliance level to the full 100% 
regulatory compliance level, there was a definite plateauing 
effect in which the programs were not increasing in quality as 
previously and in some cases, actually decreased in quality. 
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This above result was surprising and very controversial when it 
was first published in the mid 1980's. Many, if not most, 
regulatory compliance specialists did not agree with the finding. 
However, this relationship has held up in many other studies 
conducted since then and in other human service areas. It 
became the new rule in clearly demonstrating if not a decline, 
always a plateauing effect in moving from substantial to full 
compliance. Today because of all these supporting studies, the 
result is generally accepted and has influenced public 
regulatory compliance policy formulation throughout the 
world. 


REGULATORY COMPLIANCE LAW OF DIMINISHING RETURNS 


Substantial 
~~ 


i 
3 


92 94 
Regutatory Compliance Levels 


This regulatory compliance theory of diminishing returns has 
had tremendous impacts in how we have come to measure 
regulatory compliance in the licensing field. Rather than 
viewing it in a linear modality, it suggested that a more targeted, 
non-linear modality or metric might be more effective and 
efficient. Rather than focusing on full regulatory compliance it 
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suggested that a key indicator, abbreviated, or targeted 
monitoring of rules was a better approach. 


Without the regulatory compliance theory of diminishing 
returns, the focus on what has become differential monitoring 
or targeted monitoring would never have occurred. There would 
have been no need to move from always requiring full 100% 
regulatory compliance with all rules. This is a very important 
distinction and you, the learner, will see many applications and 
implications as you move through the chapters in this text. 


Moving from the Theory to the Conceptual 


Conceptually, licensing measurement is built around obviously 
licensing but there are other systems which impact on licensing 
which are demonstrated in the first licensing measurement class 
when one compares the various regulatory and non-regulatory 
systems in the Morgan Model - Methods for Achieving Quality 
Child Care. There are contractual systems, such as QRIS 
(Quality Rating and Improvement System) or other types of 
quality initiatives. These other types of quality initiatives are 
non-contractual systems, such as professional development or 
training or technical assistance systems; or accreditation 
systems. 
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Morgan Model: Methods for Achieving Quality Child Care 


Methods for Achieving Quality Child Care 
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These above systems can be integrated into a unified model 
called the arly Childhood Program Quality 
Improvement/Indicator Model or Differential Monitoring 
Logic Model and Algorithm (ECPQI2M/DMLMA) which is 
depicted in the lecture slides from the NARA Licensing 
Measurement Course that accompanies this text if you desire to 
utilize those resources and is detailed in several of the handouts. 
Since this will become the unifying framework when discussing 
licensing measurement, I would suggest that you as the learner 
spend a good deal of time reviewing those slides and handouts. 
I would think that you will want to return to them as you move 
through the upcoming chapters and classes as part of the NARA 
course to make certain you continue to understand how all the 
disparate pieces fit together into a uniformed whole. 
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Early Childhood Program Quality Indicator Model (ECPQIM) 


Yolx Yoga Yes+ Yar Yow + YD > 00 


By using the ECPQI2M/DMLMA (also see chapter 5 which 
provides a more detailed step by step guide for the development 
within a state licensing agency) it offers all the key elements to 
building an effective and efficient program monitoring system 
by integrating regulatory compliance and program quality and 
professional development systems along with differential 
monitoring's risk assessment and key indicator methodologies. 


There are readings related to professional development that are 
important components to making sure that the ECPQI2M is 
working as it should. One of the consistent key indicators deals 
with professional development/training. There are examples of 
creative and innovative ways the training can be delivered over 
the internet. Pay particular attention to the iLookOut program, 
especially to its delivery system. Check out the 
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https://RIKInstitute.com/publications/. website for these 
publications and reports, there are several articles that describe 
the program as well as its innovative cognitive mapping and 
online delivery platform. 


Program Monitoring Paradigms 


This section provides some key elements to two potential 
regulatory compliance monitoring paradigms 
(Differential/Relative versus Absolute/Full) for regulatory 
science based upon the Regulatory Compliance Theory of 
Diminishing Returns (See Figure below which depicts the 
differences between the two paradigms). 


As one will see, there is a need within regulatory science to get 
at the key measurement issues and essence of what is meant by 
regulatory compliance. There are some general principles that 
need to be dealt with such as the differences between individual 
rules and rules in the aggregate. Rules in the aggregate are not 
equal to the sum of all rules because all rules are not created nor 
administered equally. And all rules are to be adhered to, but 
there are certain rules that are more important than others and 
need to be adhered to all the time. Less important rules can be 
in substantial compliance most of the time but important rules 
must be in full compliance all of the time. 
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Regulatory Paradigms 


Absolute (Class, 1957) Relative/Differential (Fiene, 1985) 


Oo All rules are created co All rules are not 
equal. created equal. 

3 100% Compliance = a Full 100% + 
Full License. Substantial Compliance 


= Full License. 
co PC + PQ = Not Linear. 


co Selected key rules are 


co PC + PQ = Linear. 


oO All rules are 
reviewed all the time. 


reviewed all the time. 


Rules are everywhere. They are part of the human services 
landscape, economics, banking, sports, religion, transportation, 
housing, etc... Wherever one looks we are governed by rules in 
one form or another. The key is determining an effective and 
efficient modality for negotiating the path of least resistance in 
complying with a given set of rules2. It is never about more or 
less rules, it is about which rules are really productive and 
which are not. Too many rules stifle creativity, but too few rules 
lead to chaos. Determining the balance of rules is the goal and 
solution of any regulatory science paradigm. 


Differential/Relative versus  Absolute/Full Regulatory 
Compliance Paradigms: this is an important key organizational 
element in how rules are viewed when it comes to compliance. 
For example, in an absolute/full approach to regulatory 
compliance either a rule is in full compliance or not in full 
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compliance. There is no middle ground. It is black or white, no 
shades of gray as are the cases in a differential/relative 
paradigm. It is 100% or zero. In defining and viewing these two 
paradigms, this dichotomy is the organizational key element for 
this paper. In a differential/relative regulatory compliance 
paradigm full compliance is not required and emphasis on 
substantial regulatory compliance becomes the norm. 


Based upon this distinction between differential/relative and 
absolute/full regulatory compliance paradigms, what are some 
of the implications in utilizing these two respective approaches. 
Listed below are the basic implications that occur when 
selecting either of the two approaches on program monitoring 
systems: differential/relative versus absolute/full regulatory 
compliance paradigms. 


There are ten basic implications that will be addressed: 1) 
Substantial versus Monolithic. 2) Differential Monitoring 
versus One size fits all monitoring. 3) “Not all standards are 
created equal” versus “All standards are created equal”. 4) “Do 
things well” versus “Do no harm”. 5) Strength based versus 
Deficit based. 6) Formative versus Summative. 7) Program 
Quality versus Program Compliance. 8) 100-0 scoring versus 
100 or O scoring. 9) QRIS versus Licensing. 10) Non-Linear 
versus Linear. 


1) Substantial versus Monolithic: in monolithic regulatory 

compliance monitoring systems, it is one size fits all, everyone 

gets the same type of review (this is addressed in the next key 

element below) and is more typical of an absolute paradigm 

orientation. In a substantial regulatory compliance monitoring 

system, programs are monitored on the basis of their past 
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compliance history and this is more typical of a relative 
paradigm orientation. Those with high compliance may have 
fewer and more abbreviated visits/reviews while those with low 
compliance have more comprehensive visits/reviews. 


2) Differential Monitoring versus One Size Fits All Monitoring: 
how does this actually look in a program monitoring system. In 
differential monitoring (Differential/Relative Paradigm), more 
targeted or focused visits are utilized spending more time and 
resources with those problem programs and less time and 
resources with those programs that are exceptional. In the One 
Size Fits All Monitoring (Absolute/Full Paradigm), all 
programs get the same type/level of review/visit regardless of 
past performance. 


3) “Not all standards are created equal” versus “All standards 
are created equal”: when looking at standards/rules/regulations 
it is clear that certain ones have more of an impact on outcomes 
than others. For example, not having a form signed versus 
having proper supervision of clients demonstrates this 
difference. It could be argued that supervision is much more 
important to the health and safety of clients than if a form isn’t 
signed by a loved one. In a differential/relative paradigm, all 
standards are not created nor administered equally; while in an 
absolute/full paradigm of regulatory compliance, the standards 
are considered created equally and administered equally. 


4) “Do things well” versus “Do no harm” (this element is dealt 

with in the 4" chapter below as well): “doing things well” 

(Differential/Relative Paradigm) focuses on quality of services 

rather than “doing no harm” (Absolute/Full Paradigm) which 

focuses on protecting health and safety. Both are important in 
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any regulatory compliance monitoring system but a balance 
between the two needs to be found. Erring on one side of the 
equation or the other is not in the best interest of client 
outcomes. "Doing no harm" focus is on the "least common 
denominator" — the design and implementation of a monitoring 
system from the perspective of focusing on only 5% of the non- 
optimal programs ("doing no harm") rather than the 95% of the 
programs that are "doing things well". 


5) Strength based versus Deficit based: in a strength-based 
monitoring system, one looks at the glass as “half full” rather 
than as “half empty” (deficit-based monitoring system). 
Emphasis is on what the programs are doing correctly rather 
than their non-compliance with standards. A strength-based 
system is non-punitive and is not interested in catching 
programs not doing well. It is about exemplars, about excellent 
models where everyone is brought up to a new higher level of 
quality care. 


6) Formative versus Summative: differential/relative regulatory 
compliance monitoring systems are formative in nature where 
there is an emphasis on constant quality improvement and 
getting better. In  absolute/full regulatory compliance 
monitoring systems, the emphasis is on being the gate-keeper 
(more about the gate-keeper function in the next section on 
regulatory compliance/licensing and program quality) and 
making sure that decisions can be made to either grant or deny 
a license to operate. It is about keeping non-optimal programs 
from operating. 


7) Program Quality versus Program Compliance: (this element 
is dealt with in greater detail in the fourth chapter) 
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differential/relative regulatory compliance monitoring systems 
focus is on program quality and quality improvement while in 
absolute/full regulatory compliance monitoring systems the 
focus in on program compliance with rules/regulations with the 
emphasis on full, 100% compliance. 


8) “100 — 0 scoring” versus “100 or 0 scoring”: in a 
differential/relative regulatory compliance monitoring system, 
a 100 through zero (0) scoring can be used where there are 
gradients in the scoring, such as partial compliance scores. In 
an absolute/full regulatory compliance monitoring system, a 
100% or zero (0) scoring is used demonstrating that either the 
standard/rule/regulation is fully complied with or not complied 
with at all (the differences between nominal and ordinal 
measurement is dealt with in the next section on regulatory 
compliance/licensing and program quality). 


9) QRIS versus Licensing: examples of a differential/relative 
regulatory compliance monitoring system would be QRIS — 
Quality Rating and Improvement Systems. Absolute/full 
regulatory compliance systems would be state licensing 
systems. Many programs talk about the punitive aspects of the 
present human services licensing and monitoring system and its 
lack of focus on the program quality aspects in local programs. 
One should not be surprised by this because in any regulatory 
compliance system the focus is on "doing no harm" rather than 
"doing things well". It has been and continues to be the focus of 
licensing and regulations in the USA. The reason QRIS - 
Quality Rating and Improvement Systems developed in early 
care and education was to focus more on "doing things well" 
rather than "doing no harm". This is not the case in many 
Canadian Provinces and European countries in which they have 
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incorporated program quality along with specific regulatory 
requirements. 


10) Non-Linear versus Linear: the assumption in _ both 
differential/relative and absolute/full regulatory compliance 
monitoring systems is that the data are linear in nature which 
means that as compliance with rules increases, positive 
outcomes for clients increases as well. The problem is the 
empirical data does not support this conclusion. It appears from 
the data that the relationship is more non-linear where there is 
a plateau effect with regulatory compliance in which client 
outcomes increase until substantial compliance is reached but 
doesn’t continue to increase beyond this level. There appears to 
be a “sweet spot” or balancing of key rules that predict client 
outcomes more effectively than 100% or full compliance with 
all rules — this is the essence of the Theory of Regulatory 
Compliance — substantial compliance with all standards or full 
compliance with a select group of standards that predict overall 
substantial compliance and/or positive client outcomes. 


As the regulatory science and administrative fields in general 
continue to think about the appropriate monitoring systems to 
be designed and implemented, the above structure should help 
in thinking through what these measurement systems’ key 
elements should be. Both paradigms are important, in particular 
contexts, but a proper balance between the two is probably the 
best approach in designing regulatory compliance monitoring 
systems. 
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Chapter 3 
Instrument Design, Reliability and Validity, 
Statistical Methods and Databases 


This third chapter provides the learner with the key principles 
of instrument design as it relates to regulatory compliance and 
licensing measurement. As you have seen there are 
idiosyncrasies’ conceptually and theoretically and there are 
limitations as well, when it comes to instrument design. A 
major limitation with licensing data is that it is basically, 
nominal in nature. It fits the format of Yes or No responses. It 
is not ordinal in any fashion, or at least it hasn't been for the past 
50 years. In fact, it is only in the past 30 or so years that 
licensing data moved from being predominantly qualitative to 
quantitative. This change started in the 1980's with the 
publication of Instrument based program monitoring. Prior to 
that most licensing studies were written as social work case 
studies with a great deal of narrative detail but short on data 
utilization that could be used at the macro level. 


Instrument based program monitoring has its critics who are not 
overly excited about its checklist type approach. However, if a 
state is going to track where there are specific issues related to 
regulatory compliance it will be difficult unless an 
instrument/tool/checklist is not used in data collection. If there 
is continued reliance on narrative reports solely it will be 
difficult if not impossible to find any real patterns in the data. It 
is possible with the latest developments in qualitative analyses 
but it is not recommended as the sole means for tracking 
regulatory compliance. I prefer a mixed methods approach 


39 


which focuses on the strengths from both the quantitative and 
qualitative and combines both together. 


Without an instrument-based program monitoring approach it 
would be impossible to utilize the risk assessment and 
especially the licensing key indicator predictor methodologies. 
In fact, it is really a pre-requisite for designing and 
implementing a targeted monitoring or differential monitoring 
approach. 


In instrument design it is important to utilize the triangulation 
measurement strategy that looks for observation first, followed 
by record/document review, and then lastly by doing interviews 
of staff or parents. The majority of data collection should be 
through observations made in the classroom or facility. When 
observations cannot be made look for policies, files, documents 
that contain the necessary data and then lastly do interviews. 


Reliability and Validity 


This section provides the learner with the key principles of 
reliability and validity which are the mainstay of any 
measurement system. Without these two key principles we do 
not have a measurement system we can rely on. Reliability 
deals with consistency across inspectors to make certain that 
what is to be measured is measured accurately. Validity 
demonstrates that the system is working as it is supposed to. 
The results are what should be expected from a licensing or 
regulatory compliance system. 


The readings and handouts provide many examples of 
validation studies conducted in the past decade demonstrating 
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the validity and reliability of the licensing key indicator 
predictor and risk assessment methodologies (State of 
Washington and the Province of Saskatchewan (see the 
Appendix) are the best examples of these validation studies). 


Since the large influx in the use of these methodologies over the 
past couple of decades it was incumbent upon us to determine 
if these methodologies were both reliable and valid. Based upon 
these validation studies, it can now be said with a great deal of 
certainty that the methodologies do what they were intended to 
do. They statistically predict overall regulatory compliance and 
they focus on those rules that place children in greatest risk of 
morbidity or mortality keeping them safe. So, the tenet, which 
will be emphasized throughout this course "Do No Harm" is 
upheld! 


The NARA Licensing Measurement course provides the lecture 
slides where an overview and the key elements to doing 
validation studies, while the readings and handouts provide 
more of the details and the results from these studies. 
(https://www.naralicensing.org/key-indicators) 


Statistical Methods and Data Bases 


This section deals with the statistical methods used and the 
construction of the databases in licensing. As I have said 
repeatedly in my writings over the years there are many 
limitations related to licensing measurement. The statistical 
methods that can be used with licensing data are limited also, 
because we are dealing with nominal data that are severely 
skewed. Non-parametric statistics is warranted and to deal with 
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the severely skewed data, dichotomization of the data base is 
warranted. 


Dealing with data that are not normally distributed poses some 
real challenges in analyzing licensing data sets. It is paramount 
that one runs basic descriptive statistics in assessing the mean, 
standard deviation, variance, skewness, and kurtosis. It will 
help in identifying how badly the data has outliers in a very 
quantitative manner. It will also help in determining where the 
cut scores or thresholds should be for defining the high 
regulatory compliance and the low regulatory compliance 
groups. The Fiene Licensing Predictor Rules and _ their 
respective Fiene Coefficients are determined by using the phi 
coefficient in determining correlations between each rule and 
the high/low groups for regulatory compliance. This is a 
statistic used with nominal data and is used a great deal in the 
tests and measurement research literature invalidating testing 
procedures. 


The databases should be saved in .csv formatting from an Excel 
file. It is easier to import a .csv file into SPSS or PSPP which is 
the preferred statistical package for conducting these analyses. 
But definitely any statistical package can be used as well, such 
as SAS for example. Outside of generating Fiene Coefficients, 
there are no other statistical techniques that are needed in 
analyzing the database. 


The readings list (https://RIKInstitute.com/publications/) 
provides most, if not all, of the technical research notes 


generated by the Research Institute for Key Indicators. These 

tech research notes provide the latest and most up to date 

information about any changes in the methodologies for 
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generating licensing key indicator predictor rules and risk 
assessment rules. These technical research notes are really 
intended for the serious licensing research and regulatory 
scientist to delve into. They provide the specifics to the various 
statistical methodologies with specific algorithms and logic 
modeling. 


But it still important to address some of the specific statistical 
formulae pertinent to licensing and regulatory science data. For 
example, not all statistics will be relevant to licensing data 
because of its measurement limitations. Licensing data are 
nominal in nature with some instances of ordinal measurement. 
And there are other significant considerations, such as the 
skewness of the data distributions in most licensing data, non- 
linear nature of the regulatory compliance data when compared 
to quality data. So let’s start with the most pertinent statistical 
formulae to be addressed when analyzing licensing and 
regulatory compliance data. 


The first statistic we need to address is that of the skewness of 
the data. The below formula provides the basic formula for 
determining the skewness of the data distribution. Skewness is 
with out a doubt the most prominent feature of regulatory 
compliance data, so the following formula is critical as one 
thinks about analyzing their data distribution. 


mic) 


~ (n—1Kn—2) = 


43 


This formula is called the adjusted Fisher-Pearson standardized 
moment coefficient and is generally used behind the scenes in 
most software. 


Another formula you will encounter with licensing data will be 
for determining the kurtosis. The following formula provides 
the basic formula for determining the kurtosis of the licensing 
data distributions. 


Kurtosis = n * Xni (Yi - Y) 4/ (Zni (Yi - Y) 2) 2 


Another statistic that will be of importance is the variance of 
the data distribution. I have included both the population and 
sample variance formulae because in some cases we need to 
draw a sample and in other cases we have the population data. 
Variance is important with licensing data because it is very 
lacking when you really examine the data distributions which is 
not a good thing from a statistical point of view. 


Population Variance | Sample Variance 


n 


Ya -ay Ls) 


N n—-1 


o° = population variance s° = sample variance 


" ” element 


x, = value of i” element x, = value of i 


x = population mean x = sample mean 
N = population size n= sample size 
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One of the most encountered statistic is the “mean” which is the 
average of the data distribution. Unfortunately with most 
licensing data the mean is not as meaningful as the “median” 
which is the mid point of the data distribution. I have included 
both because the mean is so predominant and the median and 
quartiles are more prevalent in licensing research. The reason 
for using the median over the mean is that the licensing data 
distributions are so severely skewed. 


But the mean formula is provided below: 


Mean of Grouped Data: 


cad 
II 


mean 
frequency of each class 
mid-interval value of each class 

n total frequency 
> fx = sum of the producst of 
mid — interval values and 
their corresponding frequency 


where: X 
x 


In the majority of cases, the following formula for the median 
will be used as a better measure of central tendency and the 
average score for licensing data. Also, the data are nominal in 
measurement which means we will be collecting frequency 
data, the data are not continuous, they are very discrete. Either 
a rule or regulation is in compliance or out of compliance. 
There are no metrics in between these two extremes. Data will 
be organized and displayed in frequency tables or cross-tabs. 


Formula for calculating the median for licensing data: 
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The Median for a grouped data 


The formula for finding a median of a given data is 


rm 
5 cf 
M,, =l ts h 
( - 


| wer limit of median clas 

n= number of observations, 

cf = cumulative frequency of class preceding the median 
class 

f = frequency of median class 


h = class size [assurning class size to be equal} 


m4maths.com $ @ @ @ @m4émaths 


Also, keep in mind that the types of analyses you will be able 
to accomplish will be limited because of the nature of the data 
measurement. When it comes to looking at relationships 
between data sets you will be limited to cross-tabulations and 
the use of the chi-square statistic. I have provided the chi- 
square statistic below to be used with licensing data. Other 
statistics which require a normal distribution or a continuous 
distribution cannot and should not be used. 


A methodology that has been successful with licensing data has 
been the dichotomization of the data distribution because of the 
nominal measurement of compliance vs non-compliance. 
Generally dichotomization of data is not recommended nor 
watranted but I have found that this approach is very successful 
with licensing data distributions. 


The formula for Chi-Square which is critical in analyzing 
regulatory compliance data: 
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The Formula for Chi Square Is 
O: — E:)? 

gay CA 

where: 

c = degrees of freedom 

O = observed value(s) 

E = expected value(s) 


As one can see these statistics provide a basic grounding from 
an analytical point of view but it is limited because of the real 
limitations in the licensing data measurement characteristics. 


The formula for Key Indicators is provided here and in the last 
section of this book within the Graphs, Charts, Figures, and 
Display Section. 


Key Indicator Statistical Methodology 
a 


p = (A)(D)-(B)(C) = v(W)X)Y)(Z) 


A= High Group + Programs in Compliance on Specific Compliance Measure. 
B = High Group + Programs out of Compliance on Specific Compliance Measure. 
C = Low Group + Programs in Compliance on Specific Compliance Measure. 
D = Low Group + Programs out of Compliance on Specific Compliance Measure. 


W = Total Number of Programs in Compliance on Specific Compliance Measure. 

X = Total Number of Programs out of Compliance on Specific Compliance Measure. 
Y = Total Number of Programs in High Group. 

Z = Total Number of Programs in Low Group. 
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Chapter 4 


Regulatory Compliance and Program Quality 


This fourth chapter provides the learner with the similarities and 
differences between regulatory compliance and program 
quality. In the second chapter the regulatory compliance theory 
of diminishing returns was presented which demonstrated a 
non-linear relationship between regulatory compliance and 
program quality. In this chapter, additional concepts will be 
presented to deal with this dynamic tension between regulatory 
compliance and program quality and how we can build one 
upon the other. 


In fact, the future of licensing and regulatory compliance will 
be heavily influenced by this relationship between regulatory 
compliance and program quality. Many jurisdictions are 
attempting to build in quality to their rules/regulations. They are 
being very creative in either building separate systems (Quality 
Rating and Improvement Systems: QRIS) or attempting to build 
them right into the rules themselves in more of an ordinal 
format. 


QRIS: Quality Rating and Improvement Systems and other 
Quality Initiatives 


This section provides the learner with key examples from the 
program quality arena, such as QRIS and _ professional 
development. The ECPQI2M model presented here has these 
two systems prominently displayed along with the regulatory 
compliance or licensing system. Together they form the solid 
foundation for providing a very effective delivery system of 
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services. When these are combined with risk assessment and the 
key indicator methodologies one can add efficiency to the 
effectiveness side of the equation. The next chapter, Chapter 5, 
will get into more details about how to design an ECPQI2M 
model along with its associated logic model (DMLMA: 
Differential Monitoring Logic Model and Algorithm). 


As was mentioned in the previous section, there is a delicate 
balance between regulatory compliance and program quality. 
At all times, the ECPQI2M is to keep both regulatory 
compliance and program quality in balance, to keep health & 
safety and quality on an even keel; but as we have seen and will 
see later in this course, this balancing act can get out of kilter at 
times. 


One of the publications produced for OPRE about QRIS 
Validation is directly applicable to licensing measurement and 
has been used within this context in the validation studies that 
will be described in this course. This is an important application 
of this new framework when it comes to validation. It is not just 
for QRIS but can be applied to licensing as well. The state of 
Washington has probably some of the best examples. Please 
check out these resources and readings later on the RIKI 
website for additional examples. Look specifically for the 
Validation Framework Research Brief (Zellman & Fiene, 2012) 
published by OPRE and the state of Washington Research 
Agenda Report (Stevens & Fiene, 2015). 
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Regulatory Compliance/Licensing and Quality 


This section of the chapter will delineate the differences 
between regulatory compliance and quality. It will provide the 
essential principles and elements that clearly demonstrate the 
differences and their potential impact on program monitoring. 
Obviously, there is some overlap between this section and the 
above section dealing with regulatory compliance monitoring 
paradigms. When we think about regulatory compliance 
measurement, we are discussing licensing systems. When we 
think about quality, we are discussing Quality Rating and 
Improvement Systems (QRIS), accreditation, professional 
development, or one of the myriad quality assessment tools, 
such as the Classroom Assessment Scoring System (CLASS) or 
Environment Rating Scales (ERS’s). All these systems have 
been designed to help improve the health and safety of 
programs (licensing) to building more environmental quality 
(ERS), positive interactions amongst teachers and children 
(CLASS), enhancing quality standards (QRIS, accreditation), 
or enhancing teacher skills (professional development). 


There are ten basic principles or elements to be presented (they 
are presented in a binary fashion demonstrating differences): 1) 
“Do no harm” versus “Do good”. 2) Closed system versus Open 
system. 3) Rules versus Indicators. 4) Nominal versus Ordinal 
measurement. 5) Full versus Partial compliance. 6) Ceiling 
effect versus No Ceiling effect. 7) Gatekeeper versus Enabler. 
8) Risk versus Performance. 9) Structural versus Process 
Quality. 10) Hard versus Soft Data. 


1) Let’s start with the first principal element building off what 
was discussed in Chapter 2, “Do No Harm” versus “Do Good”. 
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In licensing, the philosophy is to do no harm, its emphasis is on 
prevention, to reduce risk to children in a particular setting. 
There is a good deal of emphasis on health and safety and not 
so much on developmentally appropriate programming. In the 
quality systems, such as QRIS, accreditation, professional 
development, Environmental Rating Scales, CLASS, the 
philosophy is to do good, its emphasis is looking at all the 
positive aspects of a setting. There is a good deal of emphasis 
on improving the programming that the children are exposed to 
or increasing the skill set of teachers, or improving the overall 
environment or interaction that children are exposed to. 


2) Closed system versus Open system. Licensing is basically a 
closed system. It has an upper limit with full compliance 
(100%) with all rules. The goal is to have all programs fully 
comply with all rules. However, the value of this assumption 
has been challenged over the years with the introduction of the 
Regulatory Compliance Theory of Diminishing Returns. With 
quality systems, they have a tendency to be more open and far 
reaching where attaining a perfect score is very difficult to come 
by. The majority of programs are more normally distributed 
where with licensing rules the majority of programs are skewed 
positively in either substantial or full compliance. It is far more 
difficult to distinguish between the really best programs and the 
mediocre programs within licensing but more successful in 
quality systems. 


3) Rules versus Indicators/Best Practices. Licensing systems 
are based around specific standards/rules/regulations that either 
are in compliance or out of compliance. It is either a program is 
in compliance or out of compliance with the specific rule. With 
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quality systems, there is more emphasis on indicators or best 
practices that are measured a bit more broadly and deal 

more with process than structure which is the case with 
licensing. It is the difference between hard and soft data as 
many legal counsels term it. There is greater flexibility in 
quality systems. With this said, if we can look at other service 
types, such as adult-residential services, there has been some 
limited success with blending structural and process elements 
but it still remains a measurement issue on the process side. 


4) Nominal versus Ordinal measurement. Licensing systems are 
nominally based measurement systems. Either you are in 
compliance or out of compliance. Nothing in-between. It is 
either a yes or no response for each rule. No maybe or partial 
compliance. With quality systems, they are generally measured 
on an ordinal level or a Likert scale. They may run from | to 3, 
or | to 5, or 1 to 7. There is more chances for variability in the 
data than in licensing which has 1 or 0 response. This increases 
the robustness of the data distribution with ordinal 
measurement. 


5) Full or None versus Gradients or Gray Area. Building off of 
the fourth element, licensing scoring is either full or not. As 
suggested in the above elements, there is no in-between 
category, no gradient or gray area. This is definitely not the case 
with quality systems in which there are gradients and 
substantial gray areas. Each best practice can be measured on a 
Likert scale with subtle gradients in improving the overall 
practice. 


6) Ceiling effect versus No Ceiling. With licensing there is 
definitely a ceiling effect because of the emphasis on full 100% 
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compliance with all rules. That is the goal of a licensing 
program, to have full compliance. With quality systems, it is 
more open-ended in which a ceiling effect is not present. 
Programs have many ways to attain excellence. 


7) Gatekeeper versus Enabler: Licensing has always been called 
a gatekeeper system. It is the entry way to providing care, to 
providing services. It is a mandatory system in which all 
programs need to be licensed to operate. In Quality systems, 
these are voluntary systems. A program chooses to participate, 
there is no mandate to participate. It is more enabling for 
programs building upon successes. There are enhancements in 
many cases. 


8) Risk versus Performance: Licensing systems are based upon 
mitigating or reducing risks to children when in out of home 
care. Quality systems are based upon performance and 
excellence where this is rewarded in their particular scoring by 
the addition of a new Star level or a Digital Badge or an 
Accreditation Certificate. 


9) Structural Quality versus Process Quality: when we think 
of structural quality, we generally think of things we can count 
easily, such as the number of children or teachers present in a 
classroom or the number of smoke alarms, etc. These are items 
that form the basis of rules within a licensing system. However, 
when we think of process quality, we generally think of things 
that are not as ease to measure, such as interactions between 
teachers and children that are warm and engaging. This is much 
more difficult to measure and generally not part of licensing 
systems but rather program quality tools, such as the ERS and 
CLASS tools. 
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10) Hard Data versus Soft Data: this dichotomy is similar to 
number 9 structure quality versus process quality but adds a 
small dimension not present in number 9. It deals with the ease 
with which legal counsel can defend a specific rule or standard 
in a court of law. Hard rules or standards are easy to measure 
while soft rules or standards are more difficult to measure or 
evaluate. Again, they fall along the continue of being structural 
versus process oriented as mentioned in 9 above. 


There has been a great deal of discussion in the early care and 
education field about the relationship between licensing, 
accreditation, QRIS, professional development, and technical 
assistance. It is important as we continue this discussion to pay 
attention to the key elements and principles in how licensing 
and these quality systems are the same and different in their 
emphases and goals, and about the implications of particular 
program monitoring paradigms and measurement strategies. 
For other regulatory systems outside the human services field, 
the same type of model can be applied positioning compliance 
and quality as a continuum one building from the other because 
I feel that with the introduction of more quality into a regulatory 
context will help to ameliorate the ceiling and plateau effect of 
diminishing returns on performance and outcomes. 
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Chapter 5 


Coordinated Program Monitoring, Differential 
Monitoring, Key Indicator, Risk Assessment, and 
Integrative Monitoring 


This chapter demonstrates the national/federal initiatives 
addressing coordinated program monitoring. There are several 
excellent reports produced by the Administration for Children 
and Families (ACF), Office of Child Care (OCC), Office of 
Planning, Research and Evaluation (OPRE), Assistant 
Secretary’s Office of Planning and Evaluation, and the Federal 
Department of Health and Human Services (HHS) which goes 
a long way in addressing this key issue. In any system where 
there are limited resources, we need to be as cost effective and 
efficient as possible. The handouts which accompany this text 
through the NARA Licensing Measurement course will provide 
you with many examples of how best to do this. These 
handouts/reports are all available of the RIKI and NARA 
websites as well as within the NARA Licensing Curriculum 
(2000). 


With a closed system and limited resources, a coordinated 
program monitoring system is critical to make certain that we 
have the necessary resources to effectively and efficiently 
protect the clients in the facilities we are mandated to license. 
The key term is "Do No Harm". The federal agency reports in 
this class will provide you with the parameters for building a 
program monitoring system that accomplishes this goal. 
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The lecture from the NARA licensing measurement course for 
this section of the chapter consists of a slide that builds upon 
Caring for Our Children Basics (CFOCB)(go _ to 
https://RIKInstitute.com) and how that publication came into 
existence. Personally, I think it is one of the most significant 
publications (CFOCB) related to early care and education 
(ECE) standards development that has ever been produced. 


CFOCB provides voluntary standards for all ECE to follow. It 
is the very essence of what coordinated program monitoring is 
all about in providing basic safeguards for all children while in 
out of home care. 


Differential Monitoring, Risk Assessment, and Key 
Indicators 


This ehandbook text has gotten into the details of differential 
monitoring, risk assessment, and the key indicator 
methodologies. We have tangentially addressed these 
methodologies throughout the text, but this chapter provides the 
step-by-step process of their development and implementation 
(see the following paragraphs and figure for ECPQIM). Also, 
there are several other publications that deal with this detail on 
the RIKI and NARA Websites. 
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Early Childhood Program Quality Indicator Model (ECPQIM) 
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The first step in utilizing the DMLMA (Differential Monitoring 
Logic Model and Algorithm) for a state is to take a close look 
at its Comprehensive Licensing Tool (CI) that it uses to collect 
violation data on all rules with all facilities in its respective 
state. If the state does not utilize a tool or checklist or does not 
review all violation data than it needs to consider these changes 
because the DMLMA is based upon an Instrument Based 
Program Monitoring System (IPM) which _ utilizes 
tools/checklists to collect data on all rules. 


The second step for the state is to compare their state’s rules 
with the National Health and Safety Performance Standards 
(Caring for Our Children)(available through 
https://RIKInstitute.com) to determine the overlap and 
coverage between the two. This is the first approach to 
validation which involves Standards review. 


57 


The third step for the state if it utilizes a Risk Assessment (RA) 
tool is to assess the relationship between this tool and Stepping 
Stones to determine the overlap and coverage between the two. 
This is a continuation of the first approach to validation which 
involves Standards review. 


The fourth step for the state is to compare the results from the 
CI with the RA tools. This step is the second approach to 
validation which involves Measures. The correlation between 
CI and RA should be at the .50 level or higher (.50+). 


In the fifth step, if a state is fortunate enough to have a QRIS — 
Quality Rating and Improvement System in place and has 
sufficient program quality (PQ) data available then they will 
have the ability to compare results from their CI tool with their 
PQ tool and validate outputs by determining the relationship 
between compliance with health and safety rules (CI) and 
program quality (PQ) measures, such as the ERS’s, CLASS, 
CDPES, etc... This is a very important step because very few 
empirical demonstrations appear in the research literature 
regarding this relationship. This step is the third approach to 
validation which involves Outputs. It would be expected that 
lower correlations (.30+) would be found between CI and PQ 
because these tools are measuring different aspects of quality 
such as health & safety versus caregiver-child interactions or 
overall classroom quality. 


The sixth step is for the state to generate a Key Indicator (KI) 

tool from the CI data base. Please see Fiene & Nixon (1985) 

and Fiene & Kroh (2000) publications available through the 

National Association for Regulatory Administration & 

Research Institute for Key Indicators Websites 
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(https://RIKInstitute/com) for a detailed explanation of the 
methodology for generating a KI tool. This step is also part of 
the second approach to validation which involves Measures. 
The correlation between the CI and KI should be very high 
(.70+) because the KI is a subset of predictor rules taken from 
the CI data base. If a state did not want to use the KI 
methodology, a direct comparison could be drawn from The 
Thirteen Indicators of Quality Child Care — this publication is 
available on the RIKI website. 


The seventh step for the state is to use the RA and KI tools 
together to determine overall compliance of facilities and how 
often and which rules will be monitored for future visits. This 
is the basic component of a Differential Monitoring (DM) 
approach and continues the second approach to validation 
(Measures). Also, this step should drive decisions within the 
technical assistance/training/professional development (PD) 
system in what resources are allocated to a particular facility. It 
would be expected that moderate correlations (.50+) would be 
found amongst RA, KI, DM, and PD. 


The eighth and final step for the state is to compare the results 
from the various monitoring tools (CI, PQ, RA, KI) with any 
child development outcome (CO) data they collect. This is a 
relatively new area and few, if any, states at this point have this 
capability on a large scale. However, as Early Learning 
Networks and Standards are developed, this will become more 
common place. This step is the fourth approach to validation 
which involves Outcomes. The correlations between CI, PQ, 
RA, KI and CO will be on the lower end (.30+) because there 
are so many other variables that impact children’s development 
other than child care facilities. 
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Key Element Definitions: CI = state or federal standards, 
usually rules or regulations that measure health and safety - 
Caring for Our Children or Head Start Performance Standards 
will be applicable here. PQ = Quality Rating and Improvement 
Systems (QRIS) standards at the state level; ERS (ECERS, 
ITERS, FDCRS), CLASS, or CDPES. RA = risk assessment 
tools/systems in which only the most critical rules/standards are 
measured. Stepping Stones is an example of this approach. KI 
= key indicators in which only predictor rules/standards are 
measured. The Thirteen Indicators of Quality Child Care is an 
example of this approach. DM = differential monitoring 
decision making in which it is determined if a program is in 
compliance or not and the number of visits/the number of 
rules/standards are ascertained from a scoring protocol. PD = 
technical assistance/training and/or professional development 
system which provides targeted assistance to the program based 
upon the DM results. CO = child outcomes which assesses how 
well the children are developing which is the ultimate goal of 
the system. 


Validation is a continuous approach and is not a once and done 
process. States should look at their monitoring systems on an 
on-going basis and make the necessary adjustments as data are 
collected and compared in order to keep program monitoring as 
cost effective and efficient. 


In the readings/handouts, the learner will find several report 
examples which provide the details of the various 
methodologies. There are more than enough examples, so pick 
the ones you are most interested in seeing. For those of you who 
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would like to see more, please go to the RIKI website and look 
under the report's webpage for additional examples. 


Just as differential monitoring helped to change the landscape 
of program monitoring in making it more sensitive to targeted 
reviewing, integrative monitoring introduces a new conceptual 
lense in how program monitoring should be done. Just as 
coordinated monitoring focuses more on the type of care 
provided, integrative monitoring focuses on the actual 
standards and how they should be formatted. It is more of a 
delicate balancing act between regulatory compliance and 
quality programming when it comes to integrative monitoring 
which is very different from coordinated monitoring which 
emphasizes the facility type. 


An interesting future research area is combining differential and 
integrative monitoring into a new approach to program 
monitoring which would emphasize risk assessment, key 
indicators, and quality programming into this new paradigm. 
Once this is done, it would be relatively easy to take those 
results and apply them within a coordinated monitoring 
approach. To a certain extent, Caring for Our Children Basics 
accomplishes this but without the increased emphasis on the 
quality programming side. 


Integrative Monitoring 


In the previous chapter, Chapter 4, the delineations between 
regulatory compliance and program quality were dichotomized 
showing how they were different and similar. This section of 
Chapter 5 introduces the notion of integrative monitoring where 
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regulatory compliance and program quality are joined together 
in a cohesive management system. 


The concept of integrative monitoring was introduced by Freer 
and Fiene (2023) in a Journal of Regulatory Science article in 
which they propose the concept along with specific constraints 
in implementing the approach. 


In the past regulatory compliance and quality programming had 
their respective silos and were not integrated within the 
monitoring function. In fact, in ECE separate quality 
initiatives, such as Quality Rating and Improvement Systems 
(QRIS), accreditation, and professional development systems 
have been developed and implemented separate from licensing 
systems in most jurisdictions. In some cases, there were 
attempts to integrate the two arenas but generally these meant 
that there was acknowledgement that both existed but standards 
remained separated and assessors measuring compliance with 
the standards or rules were in different departments. 
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ECPQIMS: Early Childhood Program Quality 
Improvement/Indicator Model Version 5 


formance A: 
Matrix, and Key Indicator 
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Professional Development Quality initiatives 


The Freer and Fiene (2023) model provides a unique means for 
combining regulatory compliance and quality programming 
into one comprehensive, effective, and efficient approach. The 
above graphic provides the basic elements in thinking about 
how all the various early care and education systems can fit 
together in a unified way. 


Freer & Fiene (2023). Regulatory compliance and quality 
programming: Constraints and opportunities for integration, 
Journal of Regulatory Science, Volume 11, Number 1, pps 1- 
10. 
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Chapter 6 


What Research Tells Us, What We Don’t Know, 
and Examples 


This ehandbook text has summarized what we know from the 
research literature about licensing measurement and regulatory 
compliance. There have been several advances in licensing 
measurement over the past couple of decades. Clearly the 
Regulatory Compliance Theory of Diminishing Returns has 
taken hold of policy development in licensing and regulatory 
administration. We have seen statutes change from requiring 
full 100% compliance in order to receive a license to operate to 
statutes that are requiring substantial regulatory compliance 
with all rules rather than full 100% compliance. Getting to 
those right rules rather than more or less rules. Caring for Our 
Children: Basics is an excellent example of this approach. 


Another example is from developing countries, especially in 
Africa as so evidence by the work of researchers in Kenya who 
have utilized the theory of regulatory compliance repeatedly in 
various industries in order to develop and promulgate effective 
and efficient regulatory policy. In fact, we have a lot to learn 
from these initiatives because they are becoming part of a new 
way of doing policy research that is on a cutting edge. 


Licensing key indicators and risk assessment rules are being 
used on a much larger scale as the differential monitoring and 
targeted monitoring approach has expanded. The latest 
Licensing Study conducted by NARA and the National Center 
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for Early Childhood Program Quality has demonstrated that the 
majority of states are using one of these approaches. 


The differential monitoring approach and its respective 
methodologies have gone through many enhancements in 
dealing with measurement and statistical nuances related to 
licensing data distributions, such as severe skewness, kurtosis, 
dichotomization of data groups, eliminating false negatives, 
limitations of nominal data analysis, moving from a nominal 
measurement scale to an ordinal measurement scale, identifying 
generic licensing key indicators, and the relationship between 
regulatory compliance & program quality (Chapters 3 & 4 
highlighted this). 


Eliminating false negatives had been a design issue in the key 
indicator methodology when substantial compliance is utilized 
for the high group dichotomization. It is not an issue when 
100% full regulatory compliance is used but there will be 
instances when substantial compliance will have to be used. 
When this occurs a revision to the original methodology and 
algorithms has to be instituted which has been outlined in a 
RIKINotes Post (January 29, 2023). Utilizing this revision 
eliminates or at least mitigates the false negative effect. 


All these above enhancements are basically dealt with and 
addressed in the RIKI Technical Research Notes found in the 
ECPQIM/DMLMA text as well as on the RIKI website 
Blog/Notes Page (https://RIKInstitute.com/blog/), or the 
National Association for Regulatory Administration website 
which was cited at the beginning of the eHandBook. The 
interested reader should find all these technical research notes 
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in one of those venues. Just look towards the end of the 
webpage to find the research notes. 


What Research Doesn't Tell Us 


So, what are the gaps in the research related to licensing 
measurement that licensing researchers and _ regulatory 
scientists should be paying attention to? This text has provided 
some of the key gaps that have been identified to date. One area 
for further research is the relationship between regulatory 
compliance and outcomes for clients. Are clients healthier and 
safer in highly compliant programs? Are we seeing fewer 
injuries in those programs of high regulatory compliance? This 
is a critical question that still needs definitive research and 
empirical evidence to confirm. 


There still needs to be additional research that continues to 
validate the rules/standards selected, the measures themselves, 
and the relationship between regulatory compliance and QRIS 
systems. There has been considerable movement in the past 
decade with validation studies being completed in many states 
and provinces and this trend needs to continue. The results to 
date definitely appear to validate all these respective 
components in that they are working as expected, but I would 
feel more confident with additional replication studies being 
completed. 


International, National, and State Examples 


This section provides us with examples mainly through the 

specific tools that have been designed by different jurisdictions 

for the differential monitoring, key indicator and risk 
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assessment methodologies described in this text. The readings 
and handouts provide many such examples which are available 
at https://RIKInstitute.com. You will find examples both from 
the USA as well as Canada. The methodologies have really 
taken off in the last decade as demonstrated by the number of 
contracts NARA has entered into with states and provinces 
throughout the United States (Montana, Michigan, Illinois, 
Indiana, Kansas, Florida, New York, Minnesota, California) 
and Canada (British Columbia, Alberta, Saskatchewan). 
Reports written describing these efforts are available on both 
the RIKI and NARA websites. See the graphic display at the 
end of this ebook in the Display, Graphs, Figures, and Charts 
Section. 


All of these jurisdictions have demonstrated a certain 
consistency when it comes to licensing key indicator predictor 
rules and risk assessment rules. There are common themes that 
have emerged over the past 4 decades. 


Here are key elements that should be present in a high-quality 
early care and education (ECE) program that any parent should 
be looking for when selecting their child care arrangement: 


e Qualified ECE teachers. 

e There is a _ stimulating and dynamic classroom 
environment where children are viewed as competent 
learners. 

e A developmentally appropriate curriculum is used 
based upon the assessed individual needs of children. 

e Opportunities for families and staff to get to know each 
other. 
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e Families receive information on their children’s 
progress regularly using a formal process. 

e arly childhood educators encourage children to 
communicate. 

e Early childhood educators encourage children to 
develop reasoning skills. 

e arly childhood educators listen attentively when 
children speak. 

e Early childhood educators speak warmly to children 


You will witness this consistency in the readings you have 
access to at https://RIKInstitute.com. Please check out the 
website because there are numerous publications and reports 
available to you. All the publications are in the public domain, 
so you are free to download them as you see fit. 


The plan is to continue validating the methodologies to make 
certain that they are keeping children healthy and safe and are 
doing no harm. That is the key element of licensing 
measurement with a focus on health and safety similar to the 
approach taken by the Nuclear Regulatory Commission (NRC) 
in keeping surrounding communities safe where nuclear power 
plants are located. 


As has been repeatedly demonstrated in this ehandbook text, 
there is a delicate balance between regulatory compliance and 
program quality (remember chapter 4). Some industries are 
more geared towards the health and safety side of the equation 
while others seek a more balanced approach of regulatory 
compliance and program quality. I have attempted to address 
both in this text and hopefully have done an equally balanced 
approach in addressing both sides of the equation. It will be 
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interesting to see how things play out as regulatory science 
continues to grow as a science and the impact of licensing 
measurement on the development of this very important 
science. 
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Chapter 7 


Future Directions/Next Steps 


This last chapter deals with where do we go from here. What 
are the next steps for licensing measurement. How do we 
combine the quantitative and the qualitative? How do we have 
a mixed methods approach? How do we combine the best 
aspects of regulatory compliance with program quality 
elements? Are there more effective ways to deal with terribly 
skewed data other than dichotomization? Does it make sense 
to move from a nominal to an ordinal measurement scale with 
regulatory compliance? All these are critical questions for the 
field of regulatory science and its accompanying licensing 
measurement. If we are truly going to build a science, we need 
to spend the requisite time on developing and implementing a 
solid scientific measurement strategy that is both reliable and 
valid. 


Two of the most critical concepts that will need addressing are 
the ceiling effect/plateauing of quality data and the variance in 
the data distribution. The ceiling effect led to the regulatory 
compliance theory of diminishing returns and has had a major 
impact on the regulatory science field. Without a solution to 
this ceiling effect, it will continue to be difficult to distinguish 
between mediocre quality in programs and high quality in 
programs. At this writing, it appears that this ceiling effect is 
an inherent characteristic of regulatory compliance systems. 
This same ceiling effect does not appear to be present in quality- 
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based program monitoring systems nor is the lack of variance 
present in those systems as well. 


Lack of variance in data distributions can and should be 
addressed by building off weighting systems that look at risk or 
performance indicators. This can be a very effective and 
efficient approach to increase the variance in a regulatory 
compliance data distribution. But it does need to be robust 
enough so that differences can be ascertained at the substantial 
regulatory compliance and the full regulatory compliance 
levels. It has always been easy to distinguish between low 
regulatory compliance and substantial regulatory compliance 
but that is not the case with substantial and full regulatory 
compliance when comparisons are made to program quality 
measures, such as the ERSs and CLASS tools. 


And these two above issues lead us to another key balancing act 
between effectiveness and efficiency. As you have seen 
throughout this ehandbook text there are several concepts that 
need to be balanced with other domains in order to be both 
effective and efficient. A good example is the use of risk 
assessment and key indicators together when designing and 
implementing a differential monitoring system. However, it is 
always possible to put this delicate balance out of sync by 
placing too much emphasis on one or the other domain. 


For example, a jurisdiction could become so efficient in 

utilizing key indicators that the tool has so few indicators that it 

begins to jeopardize the overall effectiveness of the monitoring 

system. And the other scenario is also a concern in which too 

many indicators are included on the tool in which effectiveness 

might increase but efficiency will decrease substantially. 
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Finding that correct balance is an individual study into 
attempting various strategies where clients are not placed at 
additional risk but at the same time we don’t want to jeopardize 
the overall quality of the program. 


So this leads us to a balance between regulatory compliance and 
program quality which is the essence of the last sentence of the 
previous paragraph. It is also the essence of integrative 
monitoring which is attempting to focus both on regulatory 
compliance and quality. On the surface, this sounds really easy 
to develop a balance between regulatory compliance and 
program quality; but in reality, it is difficult to pull off. The 
reason is that licensing and regulations are just not geared to 
deal with program quality. It is all about health and safety and 
focusing on risk aversion. So in moving forward this is going 
to be a difficult balancing act for most jurisdictions. 


What are some of the other issues that we will need to address 
as we move forward? Validation studies are going to be key in 
moving forward as we determine if the monitoring systems we 
have designed are working as we intended. For the one size fits 
all, this probably will not be a heavy lift; but for differential 
monitoring and integrative monitoring I think these are going 
to be heavy lifts for most jurisdictions. These systems are not 
just descriptive based systems but have inferences built in and 
this is always more difficult to validate. 


And in speaking of validation studies, a key validation study 

will be to validate the use of quality indicators which is the new 

kid on the block. Licensing key indicators have been around 

for some time, but quality key indicators have not and will need 

a good deal of research to determine what really are predictive 
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indicators. I think we do have a good start based upon the 
studies that have been done with QRIS, accreditation, and 
professional development, our major quality initiatives in the 
USA. But additional research is still needed to validate the 
initial results. It is interesting to note that the first pilot testing 
and validation of quality key indicators has occurred in a 
Canadian Province and not in the USA. Our hats are off to the 
Province of Saskatchewan for being the first to pilot test and to 
validate this new approach. 


And then there are the key measurement and statistical methods 
that need further development and refinement as it relates to 
licensing data. If we do move regulatory compliance 
attempting to balance it with program quality, there will most 
likely be experimentation in moving from a_ nominal 
measurement to an ordinal measurement scale. This idea has 
been suggested in the likes of a Regulatory Compliance Scale. 
But it is still theoretical and has not been attempted yet. But for 
the future, this will become an important area of research. 


On the statistical side, there will be the need to develop 
techniques to deal more effectively with skewed data 
distributions, false negatives, and other licensing data 
idiosyncracies. As I have said many times in this eHandBook 
licensing data are very unique. Part of this uniqueness is the 
fact that the data distributions are anything but normally 
distributed with very little variance. This is a major area of 
concern when it comes to statistical analysis and will need to be 
dealt with heads on in the coming years by regulatory scientists 
and licensing researchers. 
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This short ehandbook text is a first step in providing that 
scientific base for building a sound regulatory science, but I am 
hopeful that other licensing researchers and _ regulatory 
scientists build upon what has been presented and suggested in 
this eBook. 


For those interested in pursuing any of these topics, please don't 
hesitate to go to the RIKI Institute or the NARA websites for 
additional detailed information. Here are the pertinent websites 
for your ease of access: https://rikinstitute.com 


or https://www.naralicensing.org/key-indicators. 


Research has been going on for approximately 50 years when 
the first kernels of what a regional model for monitoring would 
look like as it related to the human services, in particular early 
care and education. I never thought it would lead to its own 
statistical methodologies and altering how licensing and 
monitoring decision making would occur. And definitely did 
not think that "differential monitoring" would be referenced in 
Federal legislation with the re-authorization of CCDBG. And 
the regulatory science field which spans all industries and 
domains concerned with the application of rules and regulations 
to our everyday existence has only coalesced over the last 20- 
25 years. 


As I said earlier, the purpose of this ehandbook was as a short 
guide for those in the regulatory science and licensing research 
arenas to get a basic understanding of licensing measurement 
and program monitoring. By starting with it and using it in 
conjunction with all the publications and materials on the RIKI 
and NARA websites as well as the NARA Licensing 
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Measurement course, it will provide an introduction to the state 
of the art regarding licensing measurement. 


Let me leave you with a Regulatory Compliance Matrix (see 
table below) which summarizes the key points in this 
ehandbook when it comes to principles of regulatory 
compliance measurement, paradigms, and the quality 
continuum; but also points us in the direction for future research 
as these are the key elements for licensing measurement and 
monitoring systems. 


The principles detail is provided in the appendix in the last 
document listed: Ten Principles of Regulatory Compliance 
Measurement. The paradigms detail can be found in chapter 2 
of this ehandbook; while the quality continuum is in chapter 4. 
There is a good deal of overlap with the 10 principles, the 10 
elements related to paradigms, and the 10 elements of the 
quality continuum. I would suggest focusing on these common 
elements and principles because they are the most significant 
pieces of the puzzle as it relates to regulatory compliance 
measurement. Taken together, these 30 principles and elements 
provide the basic parameters regulatory scientists, licensing 
researchers, and licensing administrators and policymakers 
should be focusing on when it comes to future research studies 
related to regulatory science. 
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Regulatory Compliance Matrix 


Principles Paradigms Quality Continuum 

Lack of variance Substantial vs Hard vs soft data 
monolithic 

Ceiling effect One size fits all vs Full vs partial 
differential compliance 

Difficulty between Rules are equal vs Rules vs indicators 

full and high not equal 

Nominal Do things well vs do | Do no harm vs do 

measurement no harm good 

Moving nominal to | Strength based vs Open vs closed 

ordinal deficit system 


Dichotomization Formative vs Structural vs 
summative process quality 

Lack of reliability Program quality vs | Risk vs 

and validity compliance performance 
Skewed data 100-0 vs 100 or 0 Nominal vs ordinal 
Ease between high QRIS vs licensing Gatekeeper vs 

and low enabler 

False negatives Linear vs non-linear Ceiling effect 


Although most of the research presented in this ebook is from 
the early care and education field, the many principles and 


discoveries, such as the ceiling effect, skewness of data, and 
lack of variance in data can be applied to other human service 
industries and beyond possibly in other economic fields. This 
is the particular challenge in searching out these other arenas to 
determine if these same observations will be made. 
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Appendices 


The Theory of Regulatory Compliance and Its 
Implications for Regulatory Science 

The Relationship between Early Care & Education 
Quality Initiatives and Regulatory Compliance 
Regulatory Compliance, Licensing, and Monitoring 
Measurement Principles: Rule Compliance Versus 
Rule Performance 


What is the Relationship between Regulatory 
Compliance and Complaints in a Human Services 
Licensing System? 

The Implications in Regulatory Compliance 
Measurement When Moving from Nominal to 
Ordinal Scaling 

So Which Is Better: Differential Monitoring & 
Abbreviated Inspections or Comprehensive 
Inspections? 

The Dichotomization and Bi-Polarization of the 
Matrix Data Base 

Enhanced Dichotomization Model for Generating 
Licensing Key Indicator 

The Relationship of Licensing, Head Start, Pre-K, 
QRIS, Accreditation, and Professional Development 
and their Potential Impact on Child Outcomes 
Policy Commentary: Regulatory Science 
Measurement Issues of Skewness, Dichotomization 
of Data, and Nominal versus Ordinal Data 
Measurement 
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e A Potential Reason for Skewed Regulatory 
Compliance Data Distribution 

e Data Distribution in Regulatory Science 

e Ten Principles of Regulatory Compliance 
Measurement 


These appendices provide additional detail to the above 
chapters in delving deeper into some of the key points 
made in Chapters 1-7. 


References follow the appendices and RIKINotes listing 
pertinent publications and posts related to licensing 
measurement and monitoring systems. 


Figure, Charts, Graphs, and Displays supporting the text 
in this eHandBook follow the Reference Section. This is 
followed by a Validation Report, UCM Research Abstract, 
Timeline, and Glossary of Terms. 


For additional updates before the next edition of this 
ehandbook is published, please go to either the NARA 
website https://www.naralicensing.org/key-indicators 


or the RIKI website https://rikinstitute.com/blog/ 


In addition to the NARA and RIKI websites, regular 
updating through blog posts are maintained on Medium 
and Substack for the interested reader. 
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The Theory of Regulatory Compliance* and Its 
Implications for Regulatory Science 


The theory of regulatory compliance has appeared in a series 
of articles in the Journal of Regulatory Science and its spin off 
methodologies in other journals, Child Care Quarterly, Child 
and Youth Forum, International Journal of Child Care and 
Education Policy, and Early Childhood Research Quarterly. 
The theory has had a large impact on the human services, in 
particular the early care and education field. The purpose of 
this article is to reach a larger audience that may be 
representative of some of the other regulatory areas in the 
physical sciences, medical sciences and the economic 
sciences. 


The organization of this section will first deal with the theory 
itself, explaining it in simple, non-mathematical terms and its 
implications for public policy and licensing decision making. 
Then we will delve into the implications and spin off 
methodologies of the theory, such as differential monitoring, 
risk assessment rule formulation, key predictor rules, the 
uncertainty-certainty matrix, ceiling effect, and 
dichotomization of skewed licensing data distributions. 


Regulatory science is a relatively new science appearing on 
the scene in the past 20 years. Regulatory compliance and the 
licensing of programs, industries, etc. has been around for 
quite some time. The first licensing law was passed over 100 
years ago governing orphanages in Pennsylvania. But as is 
clearly evident the science behind licensing and regulatory 
compliance lagged by many decades. Licensing grew at a 
slow pace in the human services during the twentieth century 
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and it was not until the late 1960’s to early 1970’s that human 
services began to really expand and grow in terms of the 
number of programs. Other industries grew ina 
corresponding way with most of the growth in later portions of 
the previous century. The pharmaceutical industry is a perfect 
example of this. In fact, regulatory science has really grown 
out of this need to regulate the pharmacological industry. The 
Food and Drug Administration is the leading federal agency in 
pursuing the expansion and dissemination of regulatory 
science with the establishment of national centers across the 
USA. 


Let me provide some historical context to the theory and how 
it has evolved over the past several decades based upon 
empirical evidence. The original standard paradigm when it 
came to regulatory compliance and its relationship to program 
quality was that there is a linear relationship between the two 
components. As one goes up, the other goes up in a 
corresponding way. From a public policy standpoint this 
made a great deal of sense. Any licensing agency wants to see 
increased quality of services based upon their rules and 
regulations. I will only be addressing the human services, in 
particular early care and education programs, that is where all 
the research has been done. In the future, it will be necessary 
to determine if what is being described in the human services 
applies to industries outside of this domain. 


The problem with the standard paradigm was that it was not 
based on empirical evidence but rather on expert opinion and 
anecdotal evidence, but there were no well-designed studies 
that looked at the relationship between regulatory compliance 
and program quality in any of the human services. Fast 
forward to the 1970’s as the number of early care and 
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education programs were increasing dramatically because of 
the influx of federal dollars as part of the Great Society and 
the creation of Head Start and a major expansion of child care. 
It became clear that the standard paradigm which included 
doing case studies as their major means for data collection was 
not going to be a viable measurement strategy. This ushered 
in a new form of program monitoring and data collection 
called Instrument based Program Monitoring which utilized 
checklists, tools, and instruments for their data collection and 
licensing measurement. 


Another thing that happened also in the movement from 
qualitative to quantitative measurement was that larger studies 
could be done to evaluate the relationship between regulatory 
compliance and program quality. Finally, there would be a 
chance to collect scientific data on this relationship and prove 
the linear relationship between regulatory compliance and 
program quality. When these studies were done, sure enough, 
when low levels of regulatory compliance which essentially 
means rule or regulatory violations are being found and 
comparing these data to the overall quality of the respective 
programs there was a direct linear relationship and that 
continued to be so right up to substantial regulatory 
compliance which means being 98-99% in compliance with all 
rules and regulations. However, then a very interesting 
change occurred in moving from substantial regulatory 
compliance to full (100%) regulatory compliance in which the 
respective programs did not follow the linear relationship and 
there was a plateauing or a ceiling effect in which it was 
difficult to distinguish the quality of programs that were in 
substantial vs full regulatory compliance. It was in some cases 
in subsequent studies (2010’s) which replicated these initial 


81 


studies in the 1970’s where the relationship followed more of 
a diminishing returns type of curve. Not always but definitely 
a ceiling effect was always observed in the data. 


These results obviously upset the proverbial public policy 
apple cart and the standard paradigm which was based upon a 
linear model and that licenses should only be issued to those 
programs that were in full regulatory compliance, no 
exceptions. The data did not support this claim nor the public 
policy. Substantial regulatory compliance was clearly 
demonstrating that these programs were providing the same 
level of quality care as those programs that were in full 
regulatory compliance and in some cases were doing an even 
better job of providing quality care. This is the major finding 
of the theory of regulatory compliance demonstrating these 
diminishing returns and/or ceiling effect and introduces 
substantial regulatory compliance as a licensing decision point 
rather than relying only on full 100% regulatory compliance. 
The original paradigm still holds in that regulatory compliance 
is very accurate in distinguishing between low and higher 
quality, but it is not as accurate when it comes to 
distinguishing quality at the substantial regulatory compliance 
and the full regulatory compliance levels. 


The following figure/graphic (Figure 1) depicts the 
relationship between regulatory compliance levels and 
program quality scores. This graphic is a summary depiction 
of the various studies that have been completed starting in the 
1970’s through to the 2010’s in looking at this relationship. 
The graphic also shows the relationship to several other 
concepts that will be addressed in this article, dealing with 
differential and integrative monitoring, key indicator predictor 
rules, risk assessment rules, nominal data, and 
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dichotomization of data. All these additional concepts will be 
dealt with in the following sections of this article. 


Theory of Regulatory Compliance, Key Indicators (KI), Risk Assessment (RA) 


< PQ Scores 


RC = Regulatory 
Compliance 

PQ= Program 
Kl+ Quality 

RA+ 


Dichotomization 
RC Scores 


Dichotomization 
Ki+/RA+ = High Group 
KI-/RA- = Low Group 


a Ki 
RA- 


RC Levels 
1= Full Compliance 

2 = Substantial Compliance 
3 = Med Compliance 

4 =Low Compliance 


Figure 1: Theory of Regulatory Compliance 


Let’s turn our attention to some of the spin off methodologies 
and approaches from the theory of regulatory compliance. 

The first one to consider is differential monitoring because it 

is the most significant in altering the licensing landscape in 
how programs are monitored, reviewed, and inspected. 
Differential monitoring is about focused reviews rather than a 
one size fits all approach which again was predominant in the 
standard program monitoring paradigm. Because the theory of 
regulatory compliance introduced the importance of 
substantial regulatory compliance into the new and revised 
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paradigm when it comes to program monitoring, it ushered in 
more targeted inspections or reviews which focused on key 
predictor rules or rules that placed clients at particular risk, 
more so than other rules and regulations. There was also part 
of this new paradigm the notion of reviewing programs less 
often but that was removed from the differential monitoring 
approach because all the research into program monitoring 
indicated that just reviewing the program more frequently 
brought about more positive change in regulatory compliance 
and quality. 


In Figure 2, the differential monitoring approach is depicted 
along with the definitions of each of the methodologies which 
are part of the approach. 


Risk assessment is one of the methodologies which is part of 
the differential monitoring approach. It focuses on those 
specific rules and regulations which place clients/children are 
greatest risk of morbidity or mortality. These are the rules that 
deal with supervision, hazardous materials being in locked 
cabinets, etc. Generally, jurisdictions/states/provinces can 
identify these rules through an empirical weighting approach 
where a Likert Scale is used to weight each rule or regulation 
on the basis of this morbidity and mortality dimension. Those 
rules that are determined to be highly weighted are part of the 
risk assessment rules and are to be measured in every 
differential monitoring focused review or inspection. There 
are no exceptions to this. 


Key indicator predictor rules is the other methodology which 
is part of the differential monitoring approach. Key indicator 
or predictor rules statistically predict overall regulatory 
compliance and are a very efficient metric for determining the 
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overall regulatory compliance of a facility but in a summary, 
targeted, and focused fashion without having to do a 
comprehensive inspection in looking at all the rules and 
regulations. 


Monitoring Strategies for Determining Compliance July 2014 


There are a variety of methods that many States are using, often in combination, in their monitoring and 
enforcement of licensing rules and regulations. This report explores these methods: 


= Differential Monitoring: A regulatory method for determining the frequency or depth of monitoring based on 
an assessment of a facility's history of compliance with rules; 


= Full and Abbreviated Compliance Reviews: Conducting an inspection by monitoring all rules (full review) or a 
selected set of rules (abbreviated review); 

= Risk Assessment: An approach that focuses on identifying and monitoring those rules that place children at 
greater risk of mortality or morbidity if violations or citations occur; and 


= Key Indicators: An approach that focuses on identifying and monitoring those rules that statistically predict 
compliance with all the rules. 


All Licensing Rules — Full Compliance 


Differential Monitorin 
to Visit? What is Reviewed? 


Indicators 
te u ‘ors 


Note: This graphic is adapted from a graphic developed by Dr. Richard Fiene and used in: Fiene, 
R. (2013) Differential monitoring logic model and algorithm (DMLMA: A new early childhood 
program quality indicator model (ECPQIM4)© for early care and education regulatory agencies, 
Middletown, PA: Research Institute for Key Indicators. 


The relationship among these methodologies is often confused, partly because of varying definitions. The graphic 
below explains the relationship of the methodologies with differential monitoring as the overarching approach and 
risk assessment and key indicators as types of abbreviated compliance. 


National Center on Child Care Quality Improvement, A Service of the Office of Child Care 
9300 Lee Highway, Fairfax, VA, 22031 | Phone: 877-296-2250 | Email: OCCQualityCenter@icfi.com 


Figure 2: Differential Monitoring Approaches 


Using the combined methodologies of key indicator predictor 
rules and risk assessment rules makes the differential 
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monitoring approach the most effective and efficient program 
monitoring system because it focuses on those rules where 
clients/children may be injured while at the same time 
predicting overall regulatory compliance with all the rules. It 
is the perfect balance of effectiveness and efficiency. This is 
the highly recommended way to utilize differential 
monitoring, but many jurisdictions/states/provinces use either 
the risk assessment or the key indicator methodologies, few 
are utilizing both. Hopefully this will change as the regulatory 
science field matures over the upcoming decades. 


Let’s move from the theory, program monitoring approaches 
and methodologies to the actual measurement of licensing 
data. Licensing data are at the nominal measurement level. 
This is important which will be pointed out shortly in the 
specific approach being taken here. The approach we will 
take is to use the Confusion Matrix, which is a well-known 
metric in the decision-making computer research literature and 
refocus it for regulatory science within the context of the 
definition of regulatory compliance and licensing 
measurement. It will also deal with the policy implications of 
this particular metric. It is being proposed that this new 
Uncertainty-Certainty Matrix (UCM) is a fundamental 
building block to licensing decision making. The 2 x 2 matrix 
has been written about a great deal in the development of the 
various methodologies described above and is the center piece 
for determining key indicator predictor rules, but it is also a 
core conceptual framework in licensing measurement and 
ultimately in program monitoring and reviews. 


The reason for selecting this matrix is the nature of licensing 
data, it is binary or nominal in measurement. Either a 
rule/regulation is in-compliance or out of compliance. 
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Presently most jurisdictions deal with regulatory compliance 
measurement in this nominal level or binary level. There is to 
be no gray area, this is a clear distinction in making a licensing 
decision about regulatory compliance. The UCM also takes 
the concept of Inter-Rater Reliability (IRR) a step further in 
introducing an uncertainty dimension that is very important in 
licensing decision making which is not as critical when 
calculating IRR. It is moving from an individual metric to a 
group metric involving regulatory compliance with rules. 


The key pieces to the UCM are the following: the decision (D) 
regarding regulatory compliance and actual state (S) of 
regulatory compliance. Plus (+) = In-compliance or Minus (-) 
= Out of compliance. So, let’s build the matrix: 


Table 1: Uncertainty-Certainty Matrix (UCM) Logic Model 


UCM Matrix Decision (D) Regarding Regulatory 
Logic Compliance 
(+) In Compliance (-) Not In 
Compliance 


Actual State (S)_ | (+) In Compliance Agreement Disagreement 
of 
Compliance (-) Not In Compliance Disagreement Agreement 


The above UCM matrix demonstrates when agreement and 
disagreement occur which establishes a level of certainty 
(Agreement Cells) or uncertainty (Disagreement Cells). Ina 
perfect world, there would only be agreements and no 
disagreements between the decisions made about regulatory 
compliance and the actual state of regulatory compliance. But 
from experience, this is not the case based upon reliability 
testing done in the licensing research field in which a decision 
is made regarding regulatory compliance with a specific rule 
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or regulation and then that is verified by a second observer 
who generally is considered the measurement standard. 


Disagreements raise concerns in general, but the 
disagreements are of two types: false positives and false 
negatives. A false positive is when a decision is made that a 
rule/regulation is out of compliance when it is in compliance. 
Not a good thing but its twin disagreement is worse where 
with false negatives it is decided that a rule/regulation is in 
compliance when it is out of compliance. False negatives 
need to be avoided because they place clients at extreme risk, 
more so than a false positive. False positives should also be 
avoided but it is more important to deal with the false 
negatives first before addressing the false positives. 


The next logical question after dealing with the measurement 
issues of licensing data and the fact that it is measured 
nominally is how best to deal with a data distribution which is 
severely skewed. In Figure 1, dichotomization was introduced 
in the graphic in depicting the differences between high and 
low regulatory compliance. As presented above in attempting 
to eliminate false negatives and reduce false positives, the 
same can be done by dichotomizing the licensing data 
distribution in order to accentuate the differences between low 
regulatory compliance and substantial + full regulatory 
compliance. Dichotomization of data is generally not 
recommended from a statistical point of view but because of 
the nature of licensing data being measured at the nominal 
level and being so severely skewed, it is warranted. 


Regulatory Compliance has been always approached as an all 
or none phenomenon, whether a rule is in-compliance, or it is 
not. There is no in-between or shades of gray or partial 
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compliance. This worked when the prevailing paradigm was 
that full regulatory compliance and program quality were a 
linear relationship. This was the assumption but not 
empirically verified until the later 1970’s-1980’s. When this 
assumption was put to an empirical test, it did not hold up but 
rather a curvilinear relationship between regulatory 
compliance and program quality was discovered. This upset 
the prevailing paradigm and suggested we needed a new 
approach to addressing the relationship between regulatory 
compliance and program quality as mentioned earlier in this 
article. 


It became clear after these findings in the 1970’s-80’s and 
then in the 2010’s when replication studies were completed 
that substantial regulatory compliance could not be ignored 
based upon this new theory of regulatory compliance in which 
substantial compliance acted as a “sweet spot” of best 
outcomes or results when comparing regulatory compliance 
and program quality scores. The nominal metric needed to be 
revised and more of an ordinal metric was to be its 
replacement. Because now it wasn’t just being in or out of 
compliance, but it mattered which rules were in or out of 
compliance and how they were distributed. This revised 
application involved aggregate rules and does not apply to 
individual rule scoring. The studies completed between 1970’s 
and 2010’s involved aggregate rules and not individual rules. 
To determine if the nominal to ordinal metric needs to be 
revised still needs empirical data to back this change. 


The introduction of substantial compliance into the regulatory 
compliance measurement strategy moved the field from an 
instrument-based program monitoring into a more differential 
monitoring approach. With differential monitoring this 
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approach considered which rules and how often reviews 
should be done. Also, a new Regulatory Compliance Scale 
was proposed to consider the importance of substantial 
compliance based upon the regulatory compliance theory of 
diminishing returns. As this Regulatory Compliance Scale has 
evolved within the licensing health and safety field it needs 
further revision in which program quality can be infused into 
the decision making related to individual rules. Remember that 
the original studies were concerned about rules in the 
aggregate and not individual rules. It has now become 
apparent that in dealing with the infusion of quality into rule 
formulation, a return to the individual rule approach makes the 
most sense. 


The next iteration of the Regulatory Compliance Scale will 
contain the following categories: Exceeding Full compliance, 
Full compliance, Substantial compliance, and Mediocre 
compliance to adjust for the infusion of the quality element. 
This differs slightly from the original aggregate rule 
Regulatory Compliance Scale where the categories were Full 
compliance, Substantial compliance, Mediocre compliance, 
and Low compliance where only licensing health and safety 
elements were considered (see the Table 2 below which 
depicts the regulatory compliance scales and program 
monitoring systems side by side). 


Without the theory of regulatory compliance, differential and 
integrative monitoring would not be needed because 
regulatory compliance would have had a linear relationship 
with program quality and full compliance would have been the 
ultimate goal. There would have been no need for targeted rule 
enforcement or reviews because all rules would have had an 
equal weight when it came to protecting clients and any 
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individual rule would have predicted overall compliance. But 
it “just ain’t so” as it is said. The need to make adjustments is 
brought about by the theory and it has not been the same ever 


since. 


Table 2: Regulatory Compliance Scales and Program 
Monitoring Systems 


Individual Rule 


Aggregate Rules 


Individual Rule 


Instrument based 


Differential 


Integrated 


Full Compliance 


Full Compliance 


Exceeds Compliance 


Substantial 


Full Compliance 


Mediocre 


Substantial 


Out of Compliance 


Low 


Mediocre/Low 


The above table attempts to summarize in tabular form the 
previous paragraphs in describing the relationship between 
program monitoring and licensing measurement scaling via a 
proposed Regulatory Compliance Scale. As one can see this 


moves the paradigm from a nominal to an ordinal 


measurement rubric and depicts the differences in the 
measurement focus either at the individual rule or aggregate 
rules scoring levels. It also considers the significance of 
substantial compliance given the theory of regulatory 
compliance in which substantial compliance focus is a “sweet 
spot” phenomenon as identified in the regulatory science 
research literature. It is hoped that the regulatory science field 
takes these paradigm shifts into consideration in moving 


forward with building licensing decision making systems and 


how licenses are issued to facilities. 


As a final footnote, keep in mind that the theory of regulatory 
compliance applies to the relationship between regulatory 
compliance and program quality and does not apply to 
regulatory compliance in and of itself related to health and 
safety. When dealing with regulatory compliance, full 
compliance is the ultimate goal with individual rules and in 
determining which rules are predictive rules. It is the 
preferred methodology in order to eliminate false negatives 
and decreasing false positives in making licensing decisions 
related to regulatory compliance. 


So, what are the takeaways from the theory of regulatory 
compliance and its implications for regulatory science. 


1) The theory of regulatory compliance has ushered in a 
new paradigm demonstrating the importance of 
substantial compliance and putting it on equal footing 
with full 100% regulatory compliance. 

2) Regulatory compliance will not get us to quality on its 
own, rules and regulations need an infusion of quality 
so there is the need to balance regulatory compliance 
and quality standards in any future promulgation of 
rules and regulations. 

3) How does all this fit together? An Early Childhood 
Program Quality Improvement and Indicator Model 
has been proposed to build off the results of the theory 
of regulatory compliance and to build a robust program 
monitoring system that both differentiates and 
integrates. See the following Figure 3 which provides 
a logic model for how the model would play out. 
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rules 


Yor Yoo Ym + Sars You + Yep > 00 


Figure 3: Early Childhood Program Quality Improvement 


4) 


5) 


and Indicator Model 


All the studies and research presented in this article are 
from the human services area. It will be interesting to 
see if other industries in the medical, scientific, and 
economic arenas demonstrate the same type of 
relationship between regulatory compliance in their 
respective industries and sets of rules and regulations 
and the ultimate quality of the products they produce. 
The ceiling effect, diminishing returns, plateauing all 
depict a curvilinear relationship rather than a linear 
relationship. As additional studies are completed, this 
relationship needs to be fine-tuned. Hopefully moving 
from a nominal measurement strategy to one that is 
more ordinally based via the Regulatory Compliance 
Scale will help to fine-tune that relationship. 
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6) The idiosyncratic nature of licensing data distributions 
needs to be dealt with statistically because of severe 
skewness in the data which limits the analytical frames 
that can be used. Various weighting schemes are being 
attempted in order to build in more variance in the data 
and the infusion of more quality standards into rule 
formulation should help. 

7) Hopefully, this article has given the reader the 
necessary background to understand this new paradigm 
for licensing measurement and monitoring systems 
with all its intricacies and foibles. 


References: 


Fiene, R. (2023). Licensing Measurement and Monitoring Systems: Regulatory 


Science Applied to Human Service Regulatory Administration, National 
Association for Regulatory Administration, Licensing Curriculum, Fredericksburg, 


Virginia. 


Fiene, R. (2022). Regulatory Compliance Monitoring Paradigms and the 
Relationship of Regulatory Compliance/Licensing with Program Quality: A Policy 


Commentary. Journal of Regulatory Science 10, no. 1, 1-7. 
https://doi.org/10.21423/JRS-V10A239 


Fiene, R. (2019). A treatise on theory of regulatory compliance, Journal of 
Regulatory Science, 7, no. 1, 1-3. _https://doi.org/10.21423/JRS-VO7FIENE 


Note: 


In the regulatory science research literature and in this section, both the theory of regulatory 
compliance as well as the regulatory compliance theory of diminishing returns are used 
interchangeably. 


94 


The Relationship between Early Care & Education 
Quality Initiatives and Regulatory Compliance 


Over the past couple of decades there has been many early care 
and education initiatives, such as Quality Rating and 
Improvement Systems (QRIS), Professional Development, 
Training, Technical Assistance, Accreditation, and Pre-K 
programs to just name a few. Validation and evaluation studies 
have begun to appear in the research literature, but in these 
studies, there has been few empirical demonstrations of the 
relationship between these various quality initiatives and their 
impact on regulatory compliance or a comparison to their 
respective regulatory compliance. This brief technical research 
note will provide examples of these comparisons taken from the 
Early Childhood Program Quality Improvement and Indicator 
Model (ECPQI2M) Data Base maintained at the Research 
Institute for Key Indicators (RIKIIIc). 


I have written about this back in 2014 (Fiene, 2014) in how the 
various quality initiatives were having a positive impact on the 
early care and education delivery system but at that point 
regulatory compliance data were not available. Today, in 2019, 
with many changes and developments in state data systems, this 
is no longer the case. Now it is possible to explore the 
relationships between data from the various quality initiatives 
and licensing. Several states in multiple service delivery 
systems have provided replicable findings in which I feel 
comfortable reporting out about the relationships across the 
data systems. 
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What we now know is that there is a positive and statistically 
significant relationship between regulatory compliance and 
moving up the QRIS Quality Levels. In other words, facilities 
have higher compliance in the higher QRIS Quality Levels and 
lower compliance in the lower QRIS Levels or if they do not 
participate in their state’s respective QRIS (F = 5.047 — 8.694; 
p < .0001). Other quality initiatives, such as being accredited, 
shows higher compliance with licensing rules than those 
facilities that are not accredited (t = 2.799 - 3.853; p < .005 - 
0001). 


This is a very important result clearly demonstrating the 
positive relationship between regulatory compliance and 
quality initiatives. I have some additional state data sets that I 
will add to the ECPQI2M data base and will continue to analyze 
these relationships. 


Regulatory Compliance, Licensing, and Monitoring 
Measurement Principles: Rule Compliance Versus Rule 
Performance 


The purpose of this short paper is to delineate the parameters of 
regulatory compliance, licensing and monitoring measurement 
principles (throughout this paper the term “regulatory 
compliance” will be used to encompass these principles). 
Regulatory compliance is very unique when it comes to 
measuring it because it is very different from other 
measurement systems and this impacts how one uses various 
statistical analyses. In this paper, the limitations of the 
measurement system will be highlighted with potential 


96 


solutions that have been devised over the past several decades. 
Hopefully this paper will add to the measurement and statistical 
analysis licensing research literature. It is meant for those 
agency staff who are responsible for designing regulatory 
compliance, licensing and monitoring systems. Its focus is the 
human services but the basic principles can be applied to any 
standards-based system that is based upon a compliance or 
performance model. 


The organization of this paper is as follows. First, let’s 
introduce what is included when we talk about measurement 
principles for regulatory compliance, licensing and monitoring 
systems. Second, provide examples that should be familiar to 
most individuals who have been involved in the human 
services, in particular the early care and education field. Third, 
what are the limitations of these various systems that have been 
identified in the research literature. Fourth, what are some 
potential solutions to these limitations. And, fifth, what are the 
next steps and where do we go to build reliable and valid 
measurement systems dealing with regulatory compliance, 
licensing, and program monitoring as these relate to the human 
services delivery system. 


So, what is included in this approach. I can be any rule, 
regulation, or standard based measurement system. Generally, 
these systems are focused on a nominally based system, 
sometimes they will be ordinally based. By a nominally based 
system, either the facility being assessed is in compliance with 
a particular set of rules, regulations, or standards or it is not. In 
an ordinally based system, a facility may attain a score on a 
Likert scale, such as 1 through 5 where | is non-optimal and 5 
is excellent. These types of measurement scales involve a 
performance component and are not limited to more of a 
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compliance focus as is the case with a nominally based system. 
These distinctions are important as one will see later in this 
paper when it comes to the selection of the appropriate statistics 
to measure data distributions and the subsequent analyses that 
can be undertaken. 


What are examples of these types of systems? For nominally 
based systems, just about all the licensing systems in the USA, 
Canada and beyond employ this type of measurement strategy. 
As has been said in the previous paragraph, either there is 
compliance or there is not. It is very black or white, there are 
not shades of gray. For ordinally based systems, these systems 
are a bit more diverse. Accreditation, Quality Rating and 
Improvement Systems (QRIS), the new Head Start Grantee 
Performance Management System (GPMS), the Environmental 
Rating Scales, and the CLASS are all examples of ordinally 
based systems based upon a Likert type measurement system. 
There are many others, but as a research psychologist whose 
total career (50 years) has been spent in early care and 
education, this has been the focus of my research. 


The limitations of the above systems are numerous and, in some 
ways, are difficult to find solutions. In the past, these 
measurement systems have focused more on the descriptive 
aspects of data distributions rather than attempting to be 
predictive or inferential. The first major limitation of the data 
from regulatory compliance systems is the fact that the data 
distribution is markedly skewed. What does skew data mean? 
Most data distributions are normally distributed with very few 
occurrences at the extremes with the majority of the cases in the 
middle section of the measurement scale. IQ is an example of a 
normally distributed data distribution. In a skew data 
distribution, the majority of data are at one end of the data 
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distribution, either at the positive end or the negative end of the 
distribution. With regulatory compliance data, it is at the 
positive end with the majority of facilities being in full or 100% 
compliance with the rules. Very few of the facilities are at the 
negative end of the distribution. 


What is the big deal? The big deal is that statistically we are 
limited in what we can do with the data analyses because the 
data are not normally distributed which is an assumption when 
selecting certain statistical tests. Basically, we need to employ 
non-parametric statistical analyses to deal with the data. The 
other real limitation is in the data distribution itself. It is very 
difficult to distinguish between high and mediocre facilities. It 
is very easy to distinguish between high and low performing 
facilities because of the variance between the high performing 
facilities and the low performing facilities. However, that is not 
the case between high and mediocre preforming facilities. Since 
the majority of facilities are either in full or substantial 
compliance with the rules, they are all co-mingled in a very tight 
band with little data variance. This makes it very difficult to 
distinguish differences in the facilities. And this only occurs 
with regulatory compliance data distributions. As will be 
pointed later in this paper, this is not the case with the second 
measurement system to be addressed dealing with ordinal 
measurement systems. 


There is also a confounding factor in the regulatory compliance 
data distributions which has been termed the theory of 
regulatory compliance or the law of regulatory compliance 
diminishing returns. In this theory/law, when regulatory 
compliance data are compared to program quality data, a non- 
linear relationship occurs where either the facilities scoring at 
the substantial compliance level score better than the fully 
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compliant facilities or there is a plateau effect and there is no 
significant difference between the two groups: substantial or 
fully compliant facilities when they are measured on a program 
quality scale. From a public policy stand point, this result really 
complicates how best to promulgate compliance with rules. 
This result has been found repeatedly in early care and 
education programs as well as in other human service delivery 
systems. It is conjectured that the same result will be found in 
any regulatory compliance system. 


Another limitation of regulatory compliance data is the fact that 
it is measured at a nominal level. There is no interval scale of 
measurement and usually not even an ordinal level of 
measurement. As mentioned above, either a facility is in 
compliance or not. From a statistical analytical view, again this 
limits what can be done with the data. In fact, it is probably one 
of the barriers for researchers who would like to conduct 
analyses on these data but are concerned about the robustness 
of the data and their resulting distributions. 


Let’s turn our attention to potential solutions to the above 
limitations in dealing with regulatory compliance data. One 
potential solution and this is based upon the theory of regulatory 
compliance in which substantial compliance is the threshold for 
a facility to be issued a license or certificate of compliance. 
When this public policy determination is allowed, it opens up a 
couple of alternate strategies for program monitoring and 
licensing reviews. Because of the theory of regulatory 
compliance/law of regulatory compliance diminishing returns, 
abbreviated or targeted monitoring reviews are possible, 
differential monitoring or inferential monitoring as it has been 
documented in the literature. This research literature on 


100 


differential monitoring has been dominated by two approaches: 
licensing key indicators and weighted risk assessments. 


A second solution to the above limitations deals with how we 
handle the data distribution. Generally, it is not suggested to 
dichotomize data distributions. However, when the data 
distribution is significantly skewed as it is with regulatory 
compliance, it is an appropriate adjustment to the data. By 
essentially having two groups, those facilities that are in full 
compliance and those facilities that are not in full compliance 
with the rules. In some cases, the fully compliant group can be 
combined with those facilities that are in substantial compliance 
but this should only be employed when there are not sufficient 
fully compliant facilities which is hardly never the case since 
population data and not sampled data are available from most 
jurisdictions. When data samples were drawn and the total 
number of facilities were much smaller, substantial compliant 
facilities were used as part of the grouping strategy. The 
problem in including them was that it increased the false 
negative results. With them not being included, it is possible to 
decrease and eliminate false negatives. An additional 
methodological twist is also to eliminate and not use the 
substantial compliant facilities at all in the subsequent analyses 
which again helps to accentuate the difference scores between 
the two groups of highly compliant and low compliant scoring 
facilities. 


The next steps for building valid and reliable regulatory 

compliance systems are drawing upon what has been learned 

from more ordinally based measurement systems and applying 

this measurement structure to regulatory compliance systems. 

As such, the move would be away from a strict nominally based 

measurement to more ordinal in which more of a program 
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quality element is built into each rule. By utilizing this 
paradigm shift, additional variance should be built into the 
measurement structure. So rather than having a Yes/No result, 
there would be a gradual Likert type (1-5) scale built in to 
measure “rule performance” rather than “rule compliance” 
where a “1” indicates non-compliance or a violation of the 
specific rule. A “5” would indicate excellent performance as it 
relates to the specific rule. A “3” would indicate compliance 
with the specific rule meeting the specifics of the rule but not 
exceeding it in any way. 


This paradigm shift has led to the creation of Quality Rating and 
Improvement Systems (QRIS) throughout the USA because of 
a frustration to move licensing systems to more quality focused. 
The suggestion being made here is to make this movement 
based upon the very recent developments in designing such 
systems as is the case with Head Start monitoring. Head Start 
GPMS is developing an innovative Likert based ordinal system 
which incorporates compliance and performance into their 
monitoring system. Other jurisdictions can learn from this 
development. It is not being suggested as a replacement for 
QRIS or accreditation or ERS/CLASS assessments but as a 
more seamless transition from licensing to these various 
assessments. As indicated by the theory of regulatory 
compliance and the law of regulatory compliance diminishing 
returns, this relationship between licensing and program quality 
is not linear. By having this monitoring system approach in 
place, it may be able to reintroduce more of a linear relationship 
between licensing and program quality. 
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What is the Relationship between Regulatory Compliance 
and Complaints in a Human Services Licensing System? 


Within licensing measurement and the validation of licensing 
systems it is particularly difficult to have specific outcome 
metrics that can be measured within a human services licensing 
system. The purpose of this technical research note is to propose 
a potential solution to this problem. 


Probably the most accurate measures of licensing outcomes 
focuses on improvements in the health and safety of clients 
within human services licensed facilities, such as: fewer injuries 
(safety) or higher levels of immunizations (health). Another 
measure related to client satisfaction is the number of 
complaints reported about a licensed facility by clients and the 
general public. The advantage of using complaints is that this 
form of monitoring is generally always part of an overall 
licensing system. In other words, the state/provincial licensing 
agency is already collecting these data. It is just a matter of 
utilizing these data in comparing the number of complaints to 
overall regulatory compliance. 


The author had the opportunity to have access to these data, 
complaint and regulatory compliance data in a mid-Western 
state which will be reported within this technical research note. 
There are few empirical demonstrations of this relationship 
within the licensing research literature. The following results 
are based upon a very large sample of family child care homes 
(N = 2000+) over a full year of licensing reviews. 
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The results of comparing the number of complaints and the 
respective regulatory compliance levels proved to show a rather 
significant relationship (r = .47; p < .0001). This result is the 
first step in attempting to understand this relationship as well as 
developing a methodology and analysis schema _ since 
directionality (e.g., did the complaint occur before or after the 
regulatory compliance data collection?) can play a key role in 
the relationship (this will be developed more fully in a future 
technical research note). The focus of this research note was to 
determine if any relationship existed between regulatory 
compliance and complaint data and if it is worth pursuing. 


It appears that looking more closely at the relationship between 
complaint and regulatory compliance data is warranted. It may 
provide another means of validating the fourth level of 
validation studies as proposed by Zellman and Fiene’s OPRE 
Research Brief (Zellman, G. L. & Fiene, R. (2012). Validation 
of Quality Rating and Improvement Systems for Early Care and 
Education and School-Age Care, Research-to-Policy, 
Research-to-Practice Brief OPRE 2012-29. Washington, DC: 
Office of Planning, Research and Evaluation, Administration 
for Children and Families, U.S. Department of Health and 
Human Services) in which four approaches to validation are 
delineated for Quality Rating and Improvement Systems 
(QRIS). This author has taken this framework and applied it to 
licensing systems (Fiene (2014). Validation of Georgia’s Core 
Rule Monitoring System, Georgia Department of Early Care 
and Learning) and more recently proposed as the framework for 
Washington State’s Research Agenda (Stevens & Fiene (2018). 
Validation of the Washington State’s Licensing and Monitoring 
System, Washington Department of Children, Youth, and 
Families). 
104 


The Implications in Regulatory Compliance Measurement 
When Moving from Nominal to Ordinal Scaling 


The purpose of this paper is to provide an alternate paradigm 
for regulatory compliance measurement in moving from a 
nominal to an ordinal scale measurement strategy. Regulatory 
compliance measurement is dominated by a nominal scale 
measurement system in which rules are either in compliance or 
out of compliance. There are no gradients for measurement 
within the present licensing measurement paradigm. It is very 
absolute. Either a rule is in full compliance to the letter of the 
law or the essence of the regulation or it is not. An alternate 
paradigm borrowing from accreditation and other program 
quality systems is to establish an ordinal scale measurement 
system which takes various gradients of compliance into 
account. With this alternate paradigm, it offers an opportunity 
to begin to introduce a quality element into the measurement 
schema. It also allows to take into consideration both risk and 
prevalence data which are important in rank ordering specific 
rules. 


So how would this look from a licensing decision making 
vantage point. Presently, in licensing measurement, licensing 
decisions are made at the rule level in which each rule is either 
in or out of compliance in the prevailing paradigm. Licensing 
summaries with corrective actions are generated from the 
regulatory compliance review. It is a nominal measurement 
system being based upon Yes/No responses. The alternate 
measurement paradigm I am suggesting in this paper is one that 
is more ordinal in nature where we expand the Yes/No response 
to include gradients of the particular rule. In the next paragraph, 
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I provide an example of a rule that could be measured in moving 
from a nominal to ordinal scale measurement schema. 


Rather than only measuring a rule in an all or none fashion, this 
alternate paradigm provides a more relative mode of 
measurement at an ordinal level. For example, with a 
professional development or training rule in a particular state 
which requires, let’s say, 6 hours of training for each staff 
person. Rather than having this only be 6 hours in compliance 
and anything less than this is out of compliance, let’s have this 
rule be on a relative gradient in which any amount of hours 
above the 6 hours falls into a program quality level and anything 
less than the 6 hours falls out of compliance but at a more severe 
level depending on how far below the 6 hours and how many 
staff do not meet the requirement (prevalence). Also throw in a 
specific weight which adds in a risk factor and we have a 
paradigm that is more relative rather than absolute in nature. 


From a math modeling perspective, the 1 or 0 format for a Yes 
or No response becomes -2, -1, 0, +1, +2 format. This is more 
similar to what is used in accreditation systems where 0 equals 
Compliance and -1 and -2 equals various levels of Non- 
Compliance in terms of severity and/or prevalence. The +1 and 
+2 levels equal value added to the Compliance level by 
introducing a Quality Indicator. This new formatting builds 
upon the compliance vs non-compliance dichotomy (C/NC) but 
now adds a quality indicator (QI) element. By adding this 
quality element, we may be able to eliminate or at least lessen 
the non-linear relationship between regulatory compliance with 
rules and program quality scores as measured by the 
Environmental Rating Scales (ERS) and CLASS which is the 
essence of the Theory of Regulatory Compliance (TRC). It 
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could potentially make this a more linear relationship by not 
having the data as skewed as it has been in the past. 


By employing this alternate paradigm, it is a first demonstration 
of the use of the Key Indicator Methodology in both licensing 
and quality domains. The Key Indicator Methodology has been 
utilized a great deal in licensing but in few instances in the 
program quality domain. For example, over the past five years, 
I have worked with approximately 10 states in designing 
Licensing Key Indicators but only one state with Quality Key 
Indicators from their QRIS — Quality Rating and Improvement 
System. This new paradigm would combine the use in both. It 
also takes advantage of the full ECPQI2M — Early Childhood 
Program Quality Improvement and Indicator Model by 
blending regulatory compliance with program quality 
standards. 


A major implication in moving from a nominal to an ordinal 
regulatory compliance measurement system is that it presents 
the possibility of combining licensing and quality rating and 
improvement systems into one system via the Key Indicator 
Methodology. By having licensing indicators and now quality 
indicators that could be both measured by licensing inspectors, 
there would be no need to have two separate systems but rather 
one that applies to everyone and becomes mandated rather than 
voluntary. It could help to balance both effectiveness and 
efficiency by only including those standards and rules that 
Statistically predict regulatory compliance and quality and 
balancing risk assessment by adding high risk rules. 


I will continue to develop this scale measurement paradigm 
shift in future papers but wanted to get this idea out to the 
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regulatory administration field for consideration and debate. 
This will be a very controversial proposal since state regulatory 
agencies have spent a great deal of resources on developing free 
standing QRIS which build upon licensing systems. This 
alternate paradigm builds off my Theory of Regulatory 
Compliance’s key element of relative vs absolute measurement 
and linear vs non-linear relationships. Look for additional 
information about this on my website RIKI Institute Blog - 


https://rikinstitute.com/blog/. 


So Which Is Better: Differential Monitoring & 
Abbreviated Inspections or Comprehensive Inspections? 


During 2019 and 2020, several validation studies have been or 
are being completed in the states of Washington, Indiana, and 
in the Province of Saskatchewan. These validation studies are 
determining if the key indicator and risk assessment 
methodologies are valid approaches to conducting abbreviated 
inspections in comparison to more comprehensive inspections 
in which all rules are assessed. These abbreviated inspections 
are a form of differential or targeted monitoring. This technical 
research note focuses on the empirical evidence to determine 
the efficacy of these approaches, are they better than doing 
comprehensive reviews when it comes to health and safety 
outcomes. 


When the key indicator and risk assessment methods were 
originally proposed in the 1980’s, an outcome validation study 
was completed in Pennsylvania during 1985 — 1987 by Kontos 
and Fiene to determine what impact those methods had on 
children’s development. In that original study, it was 
determined that the Child Development Program Evaluation 
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Indicator Checklist (CDPEIC) was more effective and efficient 
in predicting child development outcomes than the more 
comprehensive Child Development Program Evaluation. In 
fact, the CDPEIC and the accompanying Caregiver Observation 
Scale (COFAS) were as effective and more efficient than the 
ECERS — Early Childhood Environmental Rating Scale in that 
study. 


Fast forward to 2019 — 2020, in the province of Saskatchewan, 
Canada, and a similar study was undertaken but in this case the 
outcomes were more based upon health and safety rather than 
child development developmental outcomes. In this case, again 
the key indicator and risk assessment tool was both a more 
effective and efficient model over the more comprehensive 
inspection approach giving credence to utilizing differential 
monitoring with abbreviated inspections. 


In both of the above validation studies involving either child 
development assessment outcomes or health & safety 
outcomes, a 16 to 28% increase in effectiveness was observed 
in the outcome data. In the abbreviated or targeted inspections, 
33% of the total rules or less are used to make the determination 
of regulatory compliance. It is like having the best of both 
worlds when it comes to effectiveness (16 — 28% increase in 
outcomes) and in efficiency (66% fewer rules being used). 
These studies help to validate the use of differential monitoring 
as a viable alternative to the more comprehensive one-size-fits- 
all monitoring reviews. 
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The Dichotomization and Bi-Polarization of the Matrix 
Data Base 


This latest technical note updates the thresholds for the high and 
low groups within the key indicator matrix. This technical note 
is based upon the latest studies during the early 2015 time frame 
in which very large data distributions were available to test 
certain criteria with the key indicator methodology. Because of 
the extreme skewness present in licensing/regulatory data, 
certain statistical adjustments need to be made so that the 
analyses performed reflect the distribution of data. One of these 
Statistical adjustments is the dichotomization of data which is 
generally not suggested with the exception of very skewed data. 
Since licensing data are so skewed, this adjustment has been 
used throughout the key indicator methodology. However, an 
additional adjustment is now warranted given not only the 
skewness of data but also because of the data being nominal in 
nature. This additional adjustment I am calling the 
bipolarization of data in order to accentuate the differences 
between the high and low groups within the key indicator 
matrix. 


I have tested several data sets utilizing bi-polarization and 
found that the results are more significant with its use than 
without its use. Please keep in mind that licensing data is very 
different from other forms of data found in the early care and 
education (ECE) research literature. It is not like the ERS or 
CLASS data which is more normally distributed and lends itself 
to more parametric statistical analyses. Licensing data are 
nominal in nature and always very skewed which means that 
more non-parametric methods are warranted, such as phi 
coefficient and dichotomization of data. An example of how 
this actually works may help. 
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Licensing data are measured as either being in or out of 
compliance. There is no middle ground, it is not measured on a 
Likert scale. Therefore it is nominal in nature, either it is all 
there or it is not. Licensing data are also measured in the sense 
that all rules are created equally, in other words, they all have 
the same weight or importance, such as 1 = compliance; 0 = 
non-compliance. Being in full 100% compliance which means 
0 violations is the goal of a regulatory/licensing system. One 
does not want to see many violations of the rules because this 
will place children at risk of harm and the purpose of an early 
care and education (ECE) licensing/regulatory system is to 
reduce the potential harm to children. In the licensing 
measurement literature, this 100% compliant group is generally 
labeled or considered the high compliant group. With some 
licensing laws which allow substantial but not full 100% 
compliance with the full set of rules, it would then be allowable 
to have possibly 1 or 2 violations and still be considered in this 
high compliant group. The low compliant group has been 
generally any program that had any non-compliance or had 2 or 
more violations. When these two groups were compared to each 
individual rule utilizing the phi coefficient formula it was found 
that a more accurate approach was to accentuate or increase the 
difference between the high and low groups by eliminating the 
intervening violations in following manner: high group of 0 
violations; 1-4 violations being eliminated; 5+ violations 
defined as the low group. This additional bi-polarization of data 
helped to accentuate the differences in calculating the phi 
coefficient and provided a more sensitive key indicator tool. 


Another data distribution issue that should be addressed here 
that justifies the above cutoffs is that there is very little variance 
in licensing/regulatory data. Generally the frequency 
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distribution is 20 or less and the average set of rules is over 200 
rules. So the frequency distribution is extremely skewed within 
less than 10% of the potential data distribution. Also, the 
majority of programs are 100% in compliance with all the rules. 
And an additional complication is that the scoring of each rule 
is scored as if it had an equal risk value when in reality the rules 
can place children at either great risk to relatively little risk if 
found non-compliant. These measurement issues are very 
different than in other measurement systems such as ERS or 
CLASS. The important message to take from this is that rules 
are not a ruler, they do not measure things equally and cannot 
be analyzed or compared to other measurement systems that are 
more normally distributed. 


Although licensing is part of the program quality continuum in 
establishing basic health and safety standards for children, it is 
a system with measurement limitations that can only be 
compared on a nominal basis making several statistical 
adjustments as suggested above necessary. 


Enhanced Dichotomization Model for Generating 
Licensing Key Indicators 


The licensing key indicator methodology has been evolving 
over the past decade in making it more sensitive to the selection 
process of the specific rules to be included as key indicators. 
Some of the enhancements can occur because of state licensing 
data systems being able to provide population data rather than 
having to select sample data. Because of the nominal nature of 
licensing data and the severe skewness of the data distributions, 
non-parametric statistical approaches need to be employed in 
the analysis of the data. 
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A key component in the analysis of the licensing data 
distributions is to dichotomization of the data which is generally 
not warranted but is acceptable with very skewed data 
distributions. The dichotomization that has been most 
successful is a H25/M50/L25 distribution in which H25 
represents the High Group of regulatory compliance, M50 
which represents the Mediocre or Middle Group of regulatory 
compliance, L25 which represents the Lowest Group of 
regulatory compliance. In the past, the methodology allowed 
for full and substantial compliance within the High Group. This 
decision is no longer recommended. Rather, in order to decrease 
the number of False Negatives, it is now recommended that 
only Full (100%) regulatory compliance is used in defining the 
High Group. This eliminates the possibility of False Negatives. 


By making this above change and in using the full distribution 
of licensing data, it enhances the results for generating the 
licensing key indicator rules. For additional information on this 
modeling please see: Fiene, Richard (2018), “ECPQIM 
National Data Base”, Mendeley Data, V1. 
http://dx.doi.org/10.17632/kzk6xssx4d.1 This data base 
provides the detailed ECPQIM data distributions for the above 
changes. The enhancements increase the phi coefficients and 
reliability in either moving or not moving from abbreviated 
inspections to full comprehensive inspections. This data base 
also contains clear demonstrations of the efficacy of the 
ECPQIM — Early Childhood Program Quality Improvement 
and Indicator Model as a vehicle for improving early care and 
education programs. 
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The Relationship of Licensing, Head Start, Pre-K, QRIS, 
Accreditation, and Professional Development and their 
Potential Impact on Child Outcomes 


This short paper will provide some thoughts about the various 
public policy initiatives/systems to improve early care and 
education, such as licensing, Head Start, Pre-K, QRIS, 
accreditation, and professional development and their potential 
impact on child outcomes. Early care and education is at a major 
crossroads as a profession in attempting to determine which 
quality initiatives have the greatest impact on children. Results 
are starting to come in from early studies which may provide 
some guidance as policy makers begin making decisions about 
where to focus their limited funding resources. 


Improving early care and education programs has a long public 
policy history as we attempt to find the most cost effective and 
efficient means for attaining this lofty goal. There have been 
many ups and downs over the years where funding was 
adequate and when it was not, but our desire to accomplish this 
goal has always been front and center. Now, as a profession, we 
are at somewhat of a cross-roads in determining which of the 
many quality initiatives appear to have the greatest impact on 
children’s development. When I refer to children’s 
development, I am looking at the whole child from the 
perspective of a child’s developmental status as well as the 
child’s health and safety. 


Presently we have many quality initiatives to look at which is a 
very good thing since at times in the past we did not always 
have so many choices. Probably the one constant throughout the 
history of early care and education in the past century has been 
licensing or regulations/rule formulation. Some many argue 
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that licensing is not a quality initiative but I would suggest that 
licensing has many of the structural aspects of quality that have 
been identified in the research literature. The other quality 
initiatives I will discuss have really started and been 
implemented in the very later part of the 20th century so we are 
talking about a relatively new science when we think about 
having its intended impact on children. Also, I am talking about 
large public policy initiatives rather than highly structured, 
single focused research studies involving small samples of 
children. 


Let’s start with licensing since this system has been present for 
the longest period of time. The purpose of licensing is to act as 
the gatekeeper to the early care and education field in which 
only those providers who meet specific standards, generally 
called rules or regulations are permitted to operate and care for 
children. The rules are dominated by health and safety concerns 
with less emphasis on curriculum planning and staff-child 
interactions. The rules measure more structural aspects of 
quality than the process aspects of quality; dealing with what 
attorney’s call the “hard data” rather than the “soft data”. 


Since licensing rules allow entry into the early care and 
education field to provide services usually the rules are not 
overally stringent with the majority of providers being in high 
compliance if not full compliance with all the rules. This would 
be expected since these are basic health and safety standards. 
And in fact when one looks at compliance data, it is extremely 
skewed with the majority of providers having very high 
compliance scores with relatively few violations of the rules. 
However, this does introduce a certain difficulty in using these 
data for decision making purposes at an aggregate level because 
so many providers score at a high level it becomes increasingly 
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difficult to distinguish between the really excellent providers 
and the somewhat mediocre providers. Another way of looking 
at this skewing of the data is to term it as a plateau effect in 
which there is very little variance at the upper ends of the 
compliance spectrum. This is a major issue with skewed data 
and basic standards which is an important consideration with 
licensing but will also be an important consideration when one 
looks at the other quality initiatives to be addressed shortly. 


Because of this plateau effect with licensing data, it may explain 
much of the lack of relationships found between compliance 
with rules and any types of outcomes related to children’s 
outcomes and provider’s overall quality. However, with 
licensing data and making comparisons to children’s outcomes 
we should be looking at general health data such as 
immunization status and safety data such as the number of 
injuries at programs with varying levels of compliance with 
health and safety rules. 


A significant development over the past two decades has been 
the development of national health and safety standards with 
the publication of Caring for Our Children (CFOC3) and 
Stepping Stones (SS3). Although these standards are not 
required but are only recommended practice that provides 
guidance to states as they revise their rules, these two 
documents have been embraced by the licensing/regulatory 
administration field. Although unlikely, if not impossible, to 
comply with all the CFOC3 standards, it would be interesting 
to compare states on this set of standards which may add a good 
deal of variance to the basic health and safety data that has been 
missing with licensing rules. 
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The next system to look at is the national Head Start program. 
Out of the major programs that are national in scope, Head Start 
has a long history of providing services to low-income children 
and their families. Head Start Performance Standards are 
definitely more stringent than licensing rules but not as 
stringent as accreditation standards. Based upon Head Start’s 
more stringent standards and the additional supports that are 
part of its program, Head Start generally scores higher on 
program quality tools (e.g., CLASS or ERS) than licensed child 
care in states. 


With Head Start programs, we at times find skewing or 
plateauing of data when we compare compliance with the Head 
Start Performance Standards (HSPS) and program quality tools 
such as the CLASS. However, this is dependent upon the 
various subscales within the CLASS in which the plateauing of 
data does not occur all of the time. I think that has a lot to do 
with the HSPS being fairly stringent standards as compared to 
state licensing rules in general. 


A program that has gotten a good deal of support at the state 
level are Pre-K programs. These programs come with stricter 
standards than licensed child care with an emphasis on the 
professional development of staff. There is more concern about 
the process aspects of quality which focus more on teacher- 
child interactions. This emphasis on teacher-child interaction 
has paid off in which these programs generally are high 
performers when you compare Pre-K funded classrooms to 
licensed child care classrooms. In fact, Pre-K funding appears 
to have a positive impact on licensed child care in raising 
overall quality scores on the ECERS-R for all classrooms in 
programs that receive Pre-K funding even if some of the 
classrooms are not the direct beneficiaries of the funding. This 
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is a very significant finding because we knew that Pre-K 
funding increased the quality of care in classrooms receiving 
those funds, but now, it appears that there is a spillover effect 
to all classrooms co-located with Pre-K funded classrooms. I 
must admit that I was initially skeptical when Pre-K funding 
was first proposed because I thought it would take funding and 
the focus away from improving licensed child care at the state 
level; but it appears that the advocates for Pre-K were right in 
their assertion that Pre-K would increase the quality of all early 
care and education which includes licensed child care. 


A more recent entry into the state funding scene are QRIS 
(Quality Rating and Improvement Systems) which build upon 
licensing systems, are voluntary, and have substantial financial 
incentives for participating in this quality improvement system. 
It is too early to really determine if QRIS is having the intended 
impact because the program is so new (50% of states have a 
QRIS), and the penetration rate is usually below 50% in any 
given state (remember the system is voluntary). However, in the 
few studies done, the results are mixed. It does appear that 
programs which move up the various star levels do increase the 
quality of care they provide; but in a most recent study looking 
at child outcomes, no relationship was found between 
increasing levels of compliance with QRIS standards and how 
well children did in those programs with the exception of 
CLASS scores in which teacher-child interactions were 
measured and emphasized — here there were significant 
relationships between higher scores on the CLASS and child 
outcomes. 


Accreditation systems come in many varieties but there are only 

three that I know of in which empirical studies have been done 

to validate their systems: NAEYC, NECPA for centers and 
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NAFDC for homes. Also reliability testing has been done in 
each of these systems. Accreditation is a rigorous self-study that 
really improves programs through the self-study process. This 
should come as no surprise because we have known for some 
time that program monitoring all by itself leads to program 
improvements. Now when you couple that with technical 
assistance you see even more improvement. Accreditation is 
usually the other pillar of a QRIS system with licensing being 
the first pillar. The QRIS standards fill the gap from licensing 
to accreditation. Accreditation is a voluntary system just as in 
most cases with QRIS. However, in accreditation we are 
reaching less than 10% of the programs with the majority of 
these attaining NAEYC accreditation. NECPA and NAFDC 
have much smaller market shares. 


The last system to be addressed is the professional development 
systems that have been established in all states. This is one 
quality improvement initiative that has 100% penetration in all 
states. It is usually tied to QRIS through technical assistance 
and mentoring (coaching). When it focuses on mentoring rather 
than workshops, it has demonstrated its effectiveness in 
changing teachers behaviors in how they interact with children 
in their care in a very positive fashion. This is very important 
because the research literature is clear about the importance of 
the teacher-child interaction when it comes to child outcomes. 
Professional development runs the gamut from pre-service 
(University based programs) to in-service (training, technical 
assistance, mentoring, coaching) programming for teachers and 
directors. 


So where does this leave us when policy makers begin to try to 

determine which quality improvement initiatives should be 

invested in to start with, which to increase in funding, and 
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maybe even which ones should be defunded. I think there are 
some trends we need to begin to look at, such as the following: 


1) Having stringent and rigorous standards is very important. 
The more that we do not, the more opportunities for mediocre 
programs to score artificially higher on whatever scale that is 
used. This is evident with licensing data where the data are 
significantly skewed with a major plateau effect at the upper 
end of compliance rules/regulations. 


2) Emphasis on teacher-child interaction needs to be paramount 
in our quality improvement initiatives. Working with teachers 
through mentoring/coaching appears to be most effective in 
changing teachers’ behaviors in interacting more positively 
with children. 


3) Making sure we are measuring the right outcomes. Match 
health and safety standards with health and safety outcomes for 
children. Match developmental outcomes for children with 
standards that emphasize positive teacher-child interactions. 


4) Building upon #1 above, find what the key indicators are with 
all the data that we collect. We are spending too much time in 
looking at too many things which in many cases are simply just 
not the right things to look at. As states’ data systems become 
more sophisticated, and they are, this will be easier to do. Let’s 
begin to utilize the data we have already collected. 
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Policy Commentary: Regulatory Science Measurement 
Issues of Skewness, Dichotomization of Data, and Nominal 
versus Ordinal Data Measurement 


The purpose of this policy commentary is to provide some 
context for regulatory scientists in pursuing regulatory policy 
analysis, especially as it relates to regulatory compliance and 
human service licensing data. Regulatory scientists have dealt 
with non-parametric data in the past but in dealing with 
regulatory compliance and human service licensing data are just 
so different from previously measured data in that the nature of 
the data is nominal and extremely skewed to the point that 
several adjustments need to be made in order to analyze the 
data. 


Although the examples being referred to in this policy 
commentary are from the human services field and discipline, I 
am certain that many of the basic concepts presented will 
pertain to other disciplines and fields of study that are impacted 
by regulatory science. These concepts are not unique to a 
particular discipline but rather are unique to regulatory science 
which has particular parameters, concepts, and truths which are 
pertinent to how regulations/rules/standards are formulated and 
then implemented in various jurisdictions or disciplines. 


There are very logical reasons why regulatory compliance and 
licensing data are so extremely skewed. These data represent 
compliance with basic health and safety rules and regulations 
which provide the basic safeguards for children, youth, and 
adults while being cared for in a form of human services, such 
as child care, youth residential, or adult assisted living care. 


Very honestly a state agency would not want to find their 
regulatory compliance data being normally distributed because 
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this would be an indication that the facilities were in low 
compliance with the state's rules and regulations. Having the 
regulatory compliance data be highly negatively skewed is 
actually a good result from a public policy standpoint but not 
from a Statistical analytical standpoint. Having 50-60% of your 
scores within a three-to-five-point range when there may be as 
many as 300-400 data points leaves very little variance in the 
data. It also leads to being very difficult to distinguish between 
the high performers and the mediocre performers. This finding 
has led to a theory of regulatory compliance in which 
substantial compliance but not full compliance with all rules 
and regulations is in the best interests of the clients being served 
(Fiene, 2019). 


In the regulatory science field, this has led to public policies 
emphasizing substantial compliance in order to be a licensed 
human service facility, such as a child care center, youth 
residential program, or an adult assisted living center. The other 
aspect of regulatory compliance and licensing data for 
regulatory scientists to consider is that the data are nominal in 
measurement, either a facility is in compliance or out of 
compliance with a specific rule or regulation. There are no gray 
areas, NO measurement on an ordinal scale. 


There has been some discussion in the regulatory science field 
for the use of weighted risk assessment methodologies which 
could introduce more variance in the data based upon the 
assumption that all rules or regulations are not created equal nor 
are they administered equally (Stevens & Fiene, 2019). Another 
discussion revolves around the introduction of more program 
quality into the basic health and safety rules and regulations that 
could extend the nominal compliance determination to an 
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ordinal scale that goes beyond the basic compliance level 
(Fiene, 2018). 


These measurement idiosyncrasies of regulatory compliance 
and licensing data are presented for regulatory scientists to 
consider if they begin to analyze public policies that involve 
basic health and safety rules and regulations which are very 
different from other public policies being promulgated by state 
and national governments. For the interested reader, an 
international data base for regulatory compliance and human 
services licensing data has been established and maintained by 
the Research Institute for Key Indicators and Penn State 
University over the past 40 years at the following URL - 


(http://RIKInstitute.com) 


However, the hope is that other disciplines will begin to look at 
their data more closely to determine the natural data 
distributions and ascertain if they are equally as skewed as has 
been found in human service regulatory data. Are you 
measuring the data at a nominal level? Could they be measured 
at an ordinal level based upon a Likert scale? The data being 
referred to are regulatory compliance data which are pegged to 
specific rules/regulations/standards. It is not based upon other 
types of data collected within a regulatory frame of reference, 
such as basic demographic or descriptive data. 
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A Potential Reason for Skewed Regulatory Compliance 
Data Distributions 


One thing that is ever present with regulatory compliance data 
distributions is that they are terribly skewed. See the previous 
post which provides a definition of skewed distributions and 
their implications. This post is going to attempt to provide a 
potential answer to why the data base is skewed. 


At first, I was led to believe that potentially the skewness in the 
data was a result of the rules not being stringent enough, in other 
words, the health and safety standards were too easy to comply 
with. That could definitely be a contributing factor but this is 
not the case in all instances when one compares state human 
service rules and regulations and the Head Start Performance 
Standards. I think a much deeper structure may be operating 
that is more philosophical rather than practical. 


The philosophy of regulatory compliance and rule formulation 
is one of risk aversion. In other words, how do we mitigate risk 
that potentially increases the chances of mortality or morbidity 
in the clients being served when a specific rule is out of 
compliance. This philosophy emphasizes the elimination of a 
risk, taking something away rather than adding to it. It is 
essentially, "Do No Harm". It is interesting to note that 
generally regulatory compliance scoring is nominal in being 
either "Yes" or "No"; and a lower score is better than a higher 
score, there are fewer violations of rules. Not the way most 
assessment tools are designed. 


For example, when one looks at program quality, this system is 

based upon the open-endedness adding to rather than taking 

away. It is all about, "Do Good" rather than "Do No Harm". 

Generally when you look at the data distributions here, they are 
124 


more normally distributed without the skewed nature of 
regulatory compliance data distributions. Generally program 
quality scoring is ordinal in nature on a Likert Scale. A higher 
score is better than a lower score. Makes sense in that when you 
have more of a good thing, the higher the score. And the 
philosophy of program quality is one of improvement with 
relatively little emphasis on risk aversion. 


This is an alternate explanation to why regulatory compliance 
data distributions are so terribly skewed in comparison to other 
program quality measures. 


Data Distributions in Regulatory Science 


Data distributions in the human services as they relate to 
regulatory compliance are generally very skewed distributions 
which means that the majority of facilities being 
assessed/inspected will usually fall very close to the 100% 
compliance level. There will also be an equally large number of 
facilities that are in substantial regulatory compliance (99% - 
98% compliance levels). And then there are much fewer 
facilities that are either at a mid or low level of regulatory 
compliance (97% or lower compliance levels). One might say 
that getting a score of 97% on anything doesn't sound like it is 
mediocre or low but keep in mind we are addressing basic 
health and safety rules and not quality standards. So having 
several health and safety rules out of compliance is a big deal 
when it comes to risk assessment. It could be argued that a state 
licensing agency was not upholding its gatekeeper function by 
allowing programs to operate with such regulatory non- 
compliance. 


Why is the regulatory compliance data distribution important 
from a statistical point of view. Generally when we are dealing 
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with social science data, the data are normally distributed or 
pretty close to being normally distributed. It is a trade mark of 
a well designed assessment tool for example. So when data are 
compared to other normally distributed data, there is a good 
chance that some form of a linear relationship will be 
ascertained, albeit, not reaching statistical significance in many 
cases but linear regardless. 


When a very skewed data distribution is one of the variables as 
in the case with regulatory compliance data and it is compared 
with a normally distributed data set such as a program quality 
tool, ERS or CLASS. Well, the result is generally a non-linear 
relationship with a marked ceiling effect or plateau effect. In 
other words, the data distribution is more curvilinear than 
linear. From a practical standpoint this creates selection 
problems in the inability to identify the best programs that have 
full regulatory compliance. This can create a public policy 
nightmare in that those programs which are in substantial but 
not full regulatory compliance are as good or in some cases of 
higher quality than those programs in full regulatory 
compliance. The interesting question is does the combination of 
normally distributed data distributions with variables that have 
skewed data distributions always produce this nonlinear result?! 


And lastly, will having two variables that are skewed data 
distributions produce a more random result than if one of the 
two above conditions are present? 
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The Ten Principles of Regulatory Compliance 
Measurement 


The first principle deals with the lack of Variance in data 
distributions. Data are found to be tightly grouped at high 
compliance levels (upper 90% level). This will lead to another 
principle addressed later in this paper dealing with skewness of 
the data distribution. In fact, the majority of scores are at a full 
regulatory compliance level, in other words, 100% in 
compliance with all rules and regulations. This led to variance 
statistics showing little movement and the majority of programs 
being in very close proximity. This makes for difficult 
statistical analyses when there is little variance in the data set. 


The second principle is finding a ceiling or plateau effect in data 
distributions. It was like there was a diminishing returns effect 
as one moves from substantial regulatory compliance (upper 
90%+) to full regulatory compliance (100%) with all rules and 
regulations. This was especially true when one compares the 
regulatory compliance levels with program quality scores on 
those same programs which is addressed more in the next 
principle. 


The third principle is the difficulty distinguishing levels of 
quality between full and substantial compliance. This principle 
builds off of the previous principle dealing with a ceiling or 
plateau effect. Because so much of the data, as much as 70- 
80% of programs, are grouped so tightly at the substantial and 
high levels of regulatory compliance when one begins to go 
beyond regulatory compliance and begin to look at quality there 
is a great deal of difficulty distinguishing levels of quality. In 
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other words, the full regulatory compliant level programs are 
not necessarily the highest quality programs. 


The fourth principle is the fact that rules and regulations are 
measured at a nominal measurement level: the rules and 
regulations are either In-Compliance or Out-of-Compliance. 
The rule or regulation is measured at a “Yes” or “No” level or 
a “1” or “0” level. There are no in-between measures, no 
ordinal measurement going on. Either you got it, or you don’t. 
It is black or white, no shades of gray. It is just the nature of 
measurement when it comes to rules and regulations which are 
very different in other measurement systems. The data are very 
discrete and not continuous. They are frequency counts and not 
a ruler type of measurement. One will not find an interval level 
of measurement in any regulatory science data distribution. 


A fifth principle is attempting to move to an ordinal 
measurement level when quality is included. This principle 
builds off of the previous principle in which in some cases it 
has been suggested to add a quality component to particular 
rules or regulations. This is an interesting development and 
moves the philosophy from one of “Do no harm” to one of “Do 
things well”. It will be interesting to see how much this concept 
moves forward and changes a basic tenet in the regulatory 
science field which is more based upon health & safety, 
gatekeeper, hard data, risk aversion, and deficit based. 


The sixth principle of regulatory compliance measurement is 
the ability to dichotomize the data can be warranted because of 
the data distribution. Data dichotomization is generally not 
recommended because it accentuates differences in a data set. 
However, given the nature of regulatory compliance 
measurement being at a nominal level, fitting into a bucket 
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format, the lack of variance, and the skewness of the data 
distribution all lead to the ability to dichotomization of the data 
set. 


The seventh principle has to do with the problem with false 
negatives and positives, especially false negatives. Because of 
the data being measured in a nominal In-Compliance vs Out-of- 
Compliance dichotomy it can lead to false negatives in which 
In-Compliance decisions are made that in reality are not In- 
Compliance. False positives are a problem as well but not as 
much of a problem as false negatives. In false positives, Out- 
of-Compliance may be determined when in reality the rule or 
regulation is actually In-Compliance. This is not a good 
scenario for the provider of services, but it potentially doesn’t 
harm the client as much as when a false negative occurs. 


The eighth principle is the lack of reliability and validity testing. 
This principle builds from the previous principle in that there 
are very few examples of scientific testing of instrumentation 
and the administration of protocols to make certain that 
everything is running as it should. Because of this, it leads to 
the above problem of false positives and negatives. All 
jurisdictions need to build in regular reliability and validity 
testing to ascertain that the final decision making is within the 
ranges that are acceptable. 


The ninth principle is the ease in distinguishing levels of quality 
between low and substantial compliance. The one result that 
has been consistent over the years is the ability to see 
differences in programs that score low on_ regulatory 
compliance versus those that are at a substantial or high 
compliant level. From a licensing or regulatory administration 
point of view this is a real plus in being able to be an effective 
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gatekeeper and keeping non-optimal programs out of service. 
But as indicated in the third principle this advantage is short- 
lived as one moves up the regulatory compliance scale to 
substantial and finally to full regulatory compliance. When one 
gets to these levels it becomes increasingly difficult to 
distinguish differences in quality in those programs that are in 
substantial regulatory compliance versus those that are in full 
regulatory compliance. It appears that the regulatory 
compliance theory of diminishing returns is rearing its 
plateau/ceiling effect. The policy implications are immense 
since the assumption is that there is a linear relationship 
between program quality and regulatory compliance. How do 
we more effectively deal with this non-linear relationship in 
formulating public policy regarding licensing decision making? 


And the final tenth principle is that regulatory compliance data 
are always skewed data. The majority of programs are in 
substantial or full regulatory compliance. And in many cases, 
this can be rather severe. There generally is a long tail which 
contains some low regulatory compliant programs, but these are 
usually few in number. The data distribution just does not 
approach a normally distributed curve as we see in many other 
examples of social science data distributions. 


It is important as the regulatory science field moves forward 
that we remain cognizant of the limitations of regulatory 
compliance measurement. There are some severe limitations 
that need to be addressed (e.g., skewed data, lack of variance in 
data, ceiling effect, nominal metrics) and building in mitigation 
strategies (e.g., data dichotomization) or it will continue to lead 
to problems in our analyses (e.g., false positives and negatives). 
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The next two papers provide summaries of licensing 
measurement, regulatory compliance, and monitoring 
systems, and the importance of the theory of regulatory 
compliance; but with the lens of a much broader perspective 
going well beyond the human services and including and 
addressing other industries. 


Licensing Measurement, Regulatory Compliance and 
Monitoring Systems: Regulatory Science Applied to 
Human Services Regulatory Administration Summary 


In the realm of human services regulatory administration, 
ensuring compliance with licensing requirements is crucial for 
maintaining quality standards and safeguarding the well-being 
of individuals receiving care. As regulatory agencies strive to 
enhance their oversight and monitoring capabilities, the 
integration of measurement and monitoring systems has 
emerged as a valuable tool. 


This paper explores the significance of licensing measurement, 
regulatory compliance, and monitoring systems and delves into 
the application of regulatory science in the context of human 
services regulatory administration. It will deal with several 
issues related to this topic and expand its content beyond early 
care and education which has been more of the focus 
previously. 


Licensing measurement and monitoring systems play a crucial 
role in regulatory administration for several reasons: 
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Compliance Verification: Regulatory agencies need to ensure 
that businesses and individuals comply with specific laws, 
regulations, and standards. Licensing measurement and 
monitoring systems provide a means to verify compliance by 
collecting data and measuring various parameters. These 
systems help regulators determine whether license holders are 
meeting the required standards and taking appropriate actions 
to mitigate risks. 


Quality Assurance: Licensing measurement and monitoring 
systems contribute to quality assurance efforts by assessing the 
performance of licensed entities. They enable regulators to 
monitor the quality of services and activities associated with the 
licensing process. By establishing measurement criteria and 
tracking the relevant metrics, regulators can ensure that license 
holders maintain the desired level of quality and meet the 
expectations of consumers or the public. 


Risk Management: Many industries involve inherent risks that 
need to be managed effectively. Licensing measurement and 
monitoring systems allow regulatory agencies to assess and 
monitor the risks associated with licensed activities. By 
continuously monitoring key indicators, regulators can identify 
potential risks, deviations from safety standards, or non- 
compliance issues. This information helps regulators take 
appropriate actions to minimize risks and ensure public safety. 


Data-Driven Decision Making: Licensing measurement and 

monitoring systems generate substantial amounts of data that 

can be analyzed to make informed decisions. Regulators can 

analyze trends, patterns, and performance metrics to identify 
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areas of concern or improvement. Data-driven insights enable 
regulators to make evidence-based decisions, allocate resources 
effectively, and prioritize enforcement actions where they are 
most needed. 


Enforcement and Remediation: When non-compliance or 
deviations from regulations are identified, licensing 
measurement and monitoring systems provide evidence to 
support enforcement actions. Regulators can use the data 
collected to take appropriate enforcement measures, such as 
issuing warnings, imposing penalties, or revoking licenses. 
These systems also help in tracking the progress of remedial 
actions taken by license holders to address any identified issues 
or deficiencies. 


Transparency and Accountability: Licensing measurement and 
monitoring systems enhance transparency and accountability in 
regulatory administration. By implementing these systems, 
regulators can demonstrate their commitment to fair and 
consistent enforcement of regulations. The data collected and 
analyzed can be made accessible to the public, stakeholders, 
and policymakers, fostering trust, and allowing for external 
scrutiny of regulatory processes. 


Licensing measurement and monitoring systems are vital in 
regulatory administration as they facilitate compliance 
verification, quality assurance, risk management, data-driven 
decision making, enforcement, and accountability. These 
systems help regulators ensure that licensed entities operate 
within the set standards, mitigate risks effectively, and 
safeguard the interests of the public. 
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Regulatory Science is relevant to human services regulatory 
administration in all industries. Regulatory science is the 
scientific discipline that combines various fields, including law, 
public policy, data analysis, and risk assessment, to inform and 
guide regulatory decision-making. Measurement and 
monitoring systems are regulatory science aids in the 
development and implementation of evidence-based 
regulations and policies. 


Regulatory agencies overseeing a wide range of human 
services, such as healthcare facilities, child care centers, mental 
health institutions, and more, face several challenges in their 
oversight role. Some of the key challenges include: 


Diverse and Complex Landscape: The human services sector 
encompasses a broad range of industries, each with its unique 
complexities, regulations, and standards. Regulatory agencies 
must navigate and understand this diverse landscape to 
effectively oversee and enforce compliance. The sheer variety 
of services, settings, and stakeholders involved makes it 
challenging to develop uniform regulations and monitoring 
approaches that address the specific needs of each sector. 


Rapidly Evolving Practices and Technologies: The human 
services field is constantly evolving, with new practices, 
technologies, and treatments emerging. Regulatory agencies 
need to keep pace with these changes to ensure that the 
regulations remain relevant and up-to-date. However, this can 
be a challenging task, as it requires continuous monitoring, 
research, and adaptation of regulations to address emerging 
risks and advancements adequately. 
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Resource Constraints: Regulatory agencies often face resource 
constraints in terms of staffing, funding, and technological 
capabilities. Insufficient resources can limit their capacity to 
conduct thorough inspections, investigations, and monitoring 
activities. Additionally, limited resources may also impact the 
frequency and intensity of oversight, making it difficult to 
identify and address compliance issues effectively. 


Compliance Variability: Human services facilities and 
institutions can vary significantly in terms of size, ownership, 
resources, and compliance history. Regulatory agencies need to 
develop oversight strategies that account for these variations 
while ensuring consistent enforcement and quality standards 
across the board. Balancing the need for flexibility with the 
need for uniformity is a constant challenge for regulatory 
agencies. And this becomes increasingly complex when 
dealing with the regulatory compliance theory of diminishing 
returns/ceiling effect. 


Stakeholder Engagement and Resistance: Regulatory oversight 
often involves engaging with various stakeholders, including 
facility owners, professionals, service recipients, advocacy 
groups, and the public. These stakeholders may have different 
interests, priorities, and perspectives, leading to potential 
conflicts or resistance to regulatory measures. Balancing the 
diverse viewpoints and managing stakeholder expectations is 
essential for effective oversight. 


Data Management and Analysis: The vast amount of data 

generated by human services facilities can pose challenges in 

terms of data management, analysis, and interpretation. 

Regulatory agencies need robust systems and processes to 
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collect, store, analyze, and make sense of the data to identify 
trends, patterns, and areas of concern. The integration and 
interoperability of data systems across different sectors and 
agencies can be complex and time-consuming. 


Legal and Ethical Considerations: Regulatory agencies must 
operate within legal frameworks and adhere to ethical standards 
while overseeing human services. They need to strike a balance 
between protecting public health and safety and respecting 
individual rights and privacy. Navigating legal complexities, 
ensuring due process, and maintaining confidentiality can be 
challenging in an environment where ethical dilemmas may 
arise. 


Addressing these challenges requires a proactive and adaptive 
approach from regulatory agencies. They need to foster 
collaboration with stakeholders, invest in capacity-building 
efforts, leverage technology for efficient data management, and 
engage in continuous evaluation and improvement of their 
oversight strategies. 


Inadequate monitoring in the human services can have 
significant risks and consequences, highlighting the need for 
robust systems that ensure compliance and _ promote 
accountability. Human services encompass a wide range of 
sectors, including healthcare, social welfare, child protection, 
and criminal justice. Monitoring in these areas is essential to 
safeguard the well-being and rights of individuals, prevent 
abuses, and ensure the effective delivery of services. Here are 
some potential risks and consequences of inadequate 
monitoring: 
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Abuse and neglect: Without proper monitoring, vulnerable 
individuals may be at a higher risk of abuse, neglect, or 
exploitation. For instance, in healthcare settings, inadequate 
monitoring can lead to medical errors, mistreatment of patients, 
or substandard care. Similarly, in child protection services, 
insufficient monitoring can result in children remaining in 
abusive or neglectful environments. 


Violation of rights: Inadequate monitoring can lead to 
violations of individuals’ rights, including their civil liberties, 
privacy, and dignity. For example, in criminal justice systems, 
inadequate monitoring can result in wrongful convictions, 
excessive use of force, or violations of prisoners’ rights. In 
social welfare programs, lack of monitoring can lead to 
discrimination, improper denial of benefits, or infringement of 
recipients’ rights. 


Inefficiency and ineffective service delivery: Monitoring is 
crucial for evaluating the effectiveness and efficiency of human 
services. Without robust monitoring systems, it becomes 
challenging to identify gaps, assess performance, and make 
informed decisions for improvement. Inadequate monitoring 
may lead to wastage of resources, duplication of efforts, or the 
continuation of ineffective programs that fail to meet the needs 
of the intended beneficiaries. This is where risk assessment 
rules and key indicator rules play an important role in 
increasing the effectiveness and efficiency of the monitoring 
process by utilizing a more differential monitoring approach. 


Lack of accountability: Monitoring plays a vital role in ensuring 

accountability within human service systems. It helps identify 

and address instances of misconduct, malpractice, or non- 
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compliance with regulations and _ standards. Inadequate 
monitoring can result in a lack of transparency and 
accountability, allowing misconduct to go _ unnoticed, 
perpetrators to go unpunished, and systemic problems to 
persist. 


Loss of public trust: Inadequate monitoring erodes public trust 
in human service systems. When people perceive that their 
well-being, rights, or safety are compromised due to poor 
monitoring, it undermines their confidence in these services. 
Public trust is crucial for the effective functioning of human 
services, as it promotes cooperation, engagement, and 
participation of individuals and communities. 


To mitigate these risks and consequences, robust monitoring 
systems are essential. Such systems should include clear 
guidelines, regular inspections, audits, reporting mechanisms, 
and independent oversight bodies. They should also leverage 
technology and data analysis to enhance monitoring capabilities 
and identify patterns or anomalies. Additionally, staff training 
on monitoring protocols and the establishment of a culture of 
accountability are crucial components of an_ effective 
monitoring framework. 


Inadequate monitoring in human services poses significant risks 
and consequences. It can lead to abuse, neglect, rights 
violations, inefficiencies, lack of accountability, and loss of 
public trust. Robust monitoring systems, incorporating clear 
guidelines, regular inspections, technology, and independent 
oversight, are necessary to ensure compliance, protect 
individuals, and promote accountability within human service 
sectors. 
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The integration of measurement and monitoring systems into 
the licensing process in human services is a crucial 
development that leverages technology and data analytics to 
track, evaluate, and verify compliance with licensing standards. 
These systems provide real-time monitoring capabilities, 
enabling early detection of non-compliance, improved 
transparency, and enhanced accountability. Let's delve into the 
details of how these systems work and the benefits they bring. 


Measurement and monitoring systems in the context of human 
services licensing involve the use of advanced technologies, 
such as sensors, cameras, electronic record-keeping systems, 
and data analytics tools. These technologies are integrated into 
the licensing process to collect, analyze, and interpret relevant 
data in real-time. The aim is to ensure that organizations and 
individuals providing human services comply with the 
established licensing standards and regulations. 


One significant advantage of integrating measurement and 
monitoring systems is the early detection of non-compliance. 
With real-time monitoring, regulatory agencies can identify 
potential violations promptly. For example, if a human services 
facility is required to maintain a specific temperature range, 
sensors can continuously monitor the temperature levels. If 
there is a deviation from the acceptable range, an alert can be 
triggered, enabling swift corrective action. This early detection 
mechanism helps prevent potential risks and harm to 
individuals receiving those services. 


Moreover, these systems improve transparency by providing 
accurate and objective data. Instead of relying solely on 
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periodic inspections or self-reported information, regulatory 
agencies can access real-time data collected by the monitoring 
systems. This data-driven approach ensures a more 
comprehensive and accurate assessment of compliance with 
licensing standards. It reduces the reliance on subjective 
observations and minimizes the possibility of information gaps 
or bias. 


Furthermore, integrating measurement and monitoring systems 
enhances accountability for organizations and individuals 
providing human services. By continuously monitoring and 
recording data, these systems create an audit trail that can be 
used for accountability purposes. The collected data provides 
evidence of compliance or non-compliance with licensing 
standards, which can be used in regulatory investigations or 
legal proceedings if necessary. This level of accountability 
fosters a culture of responsibility and incentivizes compliance 
with licensing requirements. 


The benefits of these systems extend beyond regulatory 
agencies. Service providers themselves can benefit from real- 
time monitoring by gaining insights into their own operations 
and performance. By analyzing the data collected, they can 
identify areas for improvement, optimize resource allocation, 
and make evidence-based decisions to enhance the quality of 
their services. This data-driven approach supports continuous 
improvement and helps providers meet and exceed licensing 
standards. 


The integration of measurement and monitoring systems into 

the licensing process in human services offers significant 

advantages. It leverages technology and data analytics to enable 
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real-time monitoring, early detection of non-compliance, 
improved transparency, and enhanced accountability. These 
systems provide regulatory agencies with objective data to 
ensure compliance with licensing standards and promote the 
safety and well-being of individuals receiving human services. 
Simultaneously, service providers benefit from insights gained 
through data analysis, allowing them to optimize their 
operations and deliver higher quality services. 


Licensing measurement and monitoring systems in human 
services play a crucial role in ensuring compliance with 
regulations, tracking licensing activities, and monitoring the 
quality and safety of services provided. These systems typically 
consist of several key components that work together to enable 
effective measurement and monitoring. Here are the main 
components: 


Comprehensive Databases: A central database is essential for 
storing all licensing-related information, including provider 
details, facility data, licensing standards, inspection reports, and 
compliance history. These databases provide a foundation for 
data collection, analysis, and reporting. 


Example: The Child Care Licensing System (CCLS) developed 
by the Administration for Children and Families in the United 
States is a comprehensive database that tracks and manages 
child care licensing information. It allows agencies to manage 
licensing processes, track violations, and generate reports. 


Automated Data Collection Tools: Automation tools streamline 

the process of data collection by capturing information 

electronically, reducing manual effort, and improving accuracy. 
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These tools can include online application forms, electronic 
submission of documentation, and automated notifications. 


Example: The Integrated Regulatory Information System 
(IRIS) used by the California Department of Social Services 
enables online application submissions, digital document 
management, and automated notifications for licensing updates. 
It simplifies the data collection process and enhances 
efficiency. 


Risk Assessment Algorithms: Risk assessment algorithms help 
identify high-risk facilities or providers that require increased 
monitoring or intervention. These algorithms analyze various 
factors such as compliance history, complaint data, inspection 
results, and other relevant indicators to prioritize resources 
effectively. 


Example: The Risk Assessment and Management Tool (RAM) 
implemented by the Australian Government's Department of 
Health is used to assess and manage risks associated with aged 
care services. RAM employs algorithms that analyze data on 
quality indicators, complaints, and non-compliance to 
determine risk levels and allocate resources accordingly. 


Data Visualization Platforms: Data visualization platforms 
present licensing data in a user-friendly and meaningful way, 
allowing regulatory agencies to monitor trends, identify 
patterns, and make data-driven decisions. These platforms often 
include interactive dashboards, charts, and reports. 


Example: The Licensing Information System (LIS) developed 
by the Department of Health and Human Services in the state 
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of Maine provides a data visualization platform that allows 
users to generate customized reports, view interactive charts, 
and track licensing compliance trends. 


Compliance Monitoring Tools: Compliance monitoring tools 
assist in conducting inspections, audits, and other monitoring 
activities efficiently. These tools can include mobile 
applications for inspectors to collect data on-site, electronic 
checklists, and automated scheduling of inspections. 


Example: The Licensing Automation System (LAS) 
implemented by the Minnesota Department of Human Services 
offers mobile applications for licensing staff to perform 
inspections, record findings, and generate inspection reports on 
the go. It simplifies the monitoring process and improves 
accuracy. 


Overall, these components work together to create effective 
licensing measurement and monitoring systems in human 
services. By leveraging comprehensive databases, automated 
data collection tools, risk assessment algorithms, data 
visualization platforms, and compliance monitoring tools, 
regulatory agencies can enhance their oversight capabilities, 
improve efficiency, and ensure the provision of high-quality 
services while maintaining compliance with regulations. 


Licensing measurement and monitoring systems have had a 

significant impact on regulatory administration and the human 

services sector. These systems play a crucial role in enabling 

regulators to proactively identify potential risks, address 

compliance issues promptly, and ensure the safety and quality 

of services provided. In this response, we will discuss the 
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impact of these systems and provide case studies and examples 
that illustrate the positive outcomes achieved through their 
implementation. 


One of the primary benefits of licensing measurement and 
monitoring systems is their ability to provide regulators with 
real-time data and insights. These systems collect and analyze 
various metrics and indicators, allowing regulators to monitor 
the performance and compliance of service providers. By 
having access to accurate and up-to-date information, 
regulators can proactively identify potential risks and address 
them before they escalate into serious problems. 


For instance, let's consider the case of a regulatory agency 
responsible for overseeing childcare facilities. By 
implementing a licensing measurement and monitoring system, 
the agency can track key indicators such as staff-to-child ratios, 
health and safety inspections, and educational programs. If the 
system detects any deviations from the established standards, it 
can alert regulators, enabling them to intervene promptly. This 
proactive approach helps prevent incidents and ensures that 
children receive appropriate care and support. 


Another positive outcome of licensing measurement and 
monitoring systems is improved compliance management. 
These systems streamline the process of monitoring and 
assessing compliance with regulations and standards. Service 
providers can input data directly into the system, reducing the 
administrative burden and ensuring accuracy. Regulators can 
then use this data to identify patterns, assess compliance levels, 
and take appropriate actions if non-compliance is detected. 
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For example, let's consider the case of a regulatory agency 
overseeing healthcare facilities. With a licensing measurement 
and monitoring system in place, the agency can track indicators 
such as medication errors, infection rates, and patient 
satisfaction scores. If the system identifies a healthcare facility 
with consistently high medication error rates, regulators can 
conduct targeted inspections and work closely with the facility 
to implement corrective measures. This proactive approach not 
only improves patient safety but also helps service providers 
enhance the quality of care they deliver. 


Furthermore, licensing measurement and monitoring systems 
contribute to transparency and accountability in the human 
services sector. These systems provide a centralized platform 
where regulators, service providers, and the public can access 
information about licensing status, compliance records, and 
performance metrics. By promoting transparency, these 
systems help build trust among stakeholders and empower 
individuals to make informed decisions about service providers. 


For instance, in the context of elder care services, a licensing 
measurement and monitoring system can provide a public 
database that includes information on the licensing status of 
assisted living facilities, compliance records related to safety 
standards, and ratings based on resident satisfaction surveys. 
This enables families and individuals seeking care for their 
loved ones to make informed choices and select facilities that 
meet their specific needs. 


Licensing measurement and monitoring systems have had a 

transformative impact on regulatory administration and the 

human services sector. These systems enable regulators to 
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proactively identify potential risks, address compliance issues 
promptly, and ensure the safety and quality of services 
provided. Through case studies and examples, we have seen 
how these systems have improved oversight in childcare, 
healthcare, and elder care, leading to positive outcomes such as 
enhanced safety, improved compliance, and _ increased 
transparency. The implementation of such systems has the 
potential to further strengthen regulatory efforts and promote 
the well-being of individuals receiving human services. 


Licensing measurement and monitoring systems can present 
various challenges and considerations, including privacy 
concerns, data security, resource constraints, and the need for 
ongoing system updates and maintenance. Addressing these 
challenges is crucial to ensure the effective implementation and 
operation of these systems. Additionally, collaboration between 
regulatory agencies, stakeholders, and technology providers is 
essential to overcome these challenges and maximize the 
benefits of these systems. 


Privacy concerns: Measurement and monitoring systems often 
involve the collection and analysis of sensitive data, such as 
personal information or proprietary business data. It is 
important to establish robust privacy policies and legal 
frameworks to protect individuals’ privacy rights and ensure 
compliance with relevant data protection regulations. 
Implementing anonymization techniques, data minimization 
principles, and obtaining appropriate consent can help mitigate 
privacy concerns. 


Data security: The storage, transmission, and analysis of 
measurement and monitoring data require robust security 
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measures to prevent unauthorized access, data breaches, or 
cyber-attacks. Encryption, access controls, regular security 
audits, and adherence to industry best practices can help 
safeguard the data and maintain its integrity and confidentiality. 


Resource constraints: Licensing measurement and monitoring 
systems can pose financial and logistical challenges, 
particularly for smaller organizations or developing countries 
with limited resources. These systems may require substantial 
investments in infrastructure, equipment, and skilled personnel. 
Adequate funding mechanisms, public-private partnerships, 
and capacity-building initiatives can help address resource 
constraints and ensure broader access to these systems. 


Ongoing system updates and maintenance: Measurement and 
monitoring systems must be regularly updated to keep pace 
with evolving technologies, regulatory requirements, and 
scientific advancements. This necessitates ongoing 
maintenance, software updates, calibration, and quality control 
procedures. Collaboration between regulatory agencies, 
technology providers, and stakeholders is crucial to establish 
effective mechanisms for system maintenance, ensuring that the 
systems remain accurate, reliable, and up-to-date. 


Collaboration between regulatory agencies, stakeholders, and 
technology providers: Overcoming the challenges associated 
with licensing measurement and monitoring systems requires a 
collaborative approach. Regulatory agencies should engage in 
constructive dialogues with stakeholders, including industry 
representatives, environmental organizations, and community 
groups. Collaboration can help address concerns, establish 
common standards, and promote transparency and 
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accountability. Technology providers can contribute by 
developing user-friendly and interoperable systems that meet 
regulatory requirements while minimizing the burden on end- 
users. 


Collaboration among regulatory agencies, stakeholders, and 
technology providers is critical to ensure the successful 
implementation of measurement and monitoring systems. By 
working together, these entities can develop robust policies, 
address privacy concerns, enhance data security, allocate 
necessary resources, and establish mechanisms for ongoing 
system updates and maintenance. This collaborative approach 
will maximize the effectiveness of these systems in monitoring 
and safeguarding various aspects of public health, 
environmental quality, and regulatory compliance. 


Licensing measurement and monitoring systems play a crucial 
role in human services regulatory administration by ensuring 
compliance, enhancing service quality, and _ protecting 
individuals receiving care. Integrating regulatory science 
principles into licensing processes further strengthens these 
benefits. 


One significant aspect of licensing measurement and 
monitoring systems is their ability to promote compliance. 
These systems provide a standardized framework for evaluating 
and assessing the compliance of service providers with 
established regulations and standards. By implementing these 
systems, regulatory authorities can systematically track and 
measure compliance levels, identify areas of non-compliance, 
and take appropriate actions to rectify any deficiencies. This 
helps maintain a high level of accountability among service 
148 


providers, ensuring they adhere to the required standards and 
regulations. 


Moreover, integrating regulatory science principles into 
licensing processes brings several advantages. Regulatory 
science applies scientific knowledge and methodologies to 
inform regulatory decision-making. By incorporating these 
principles into licensing, regulators can leverage evidence- 
based approaches to establish standards, design measurement 
tools, and set performance benchmarks. This approach 
promotes objectivity, transparency, and consistency in the 
licensing process, ensuring that decisions are based on sound 
scientific evidence rather than subjective judgment. 


Another key benefit is the potential for improved service 
quality. Licensing measurement and monitoring systems enable 
regulators to gather comprehensive data on service providers’ 
performance, outcomes, and service quality indicators. This 
information allows for a thorough assessment of service 
delivery, identifying strengths and weaknesses in the system. 
By analyzing this data, regulators can provide feedback, 
guidance, and support to service providers, fostering continuous 
improvement in service quality. This leads to better outcomes 
for individuals receiving care and enhances overall service 
provision within the human services sector. 


Furthermore, licensing measurement and monitoring systems 

are instrumental in protecting the well-being of individuals 

receiving care. These systems help identify potential risks, such 

as violations of safety protocols or instances of abuse or neglect. 

By closely monitoring service providers, regulators can swiftly 

respond to any issues, take necessary corrective actions, and 
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ensure the safety and well-being of vulnerable populations. 
Regular monitoring also acts as a deterrent, encouraging service 
providers to maintain high standards and comply with 
regulations to avoid penalties or sanctions. 


Looking ahead, the field of regulatory science and measurement 
and monitoring systems is continually evolving. Advances in 
technology, data analytics, and artificial intelligence present 
opportunities for further advancements in these systems. For 
example, the integration of real-time data collection and 
analysis can enhance the effectiveness and efficiency of 
monitoring processes. Predictive analytics and risk assessment 
models can help regulators proactively identify potential areas 
of concern and allocate resources accordingly. Additionally, the 
incorporation of feedback from individuals receiving care and 
other stakeholders can further refine measurement systems, 
ensuring they capture the most relevant and meaningful 
indicators of service quality. 


In conclusion, licensing measurement and monitoring systems 
are vital components of human _ services regulatory 
administration. By integrating regulatory science principles, 
these systems promote compliance, improve service quality, 
and protect individuals receiving care. As regulatory science 
continues to evolve, the potential for further advancements in 
measurement and monitoring systems is promising, enabling 
regulators to better fulfill their mandate of safeguarding the 
well-being of vulnerable populations. 
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Importance of the Theory of Regulatory Compliance 
Introduction 


Regulatory compliance refers to the process by which 
individuals, organizations, or entities adhere to and fulfill the 
requirements set forth by relevant laws, regulations, and 
industry standards. It involves ensuring that policies, 
procedures, and practices align with the specific legal and 
regulatory frameworks applicable to a particular industry or 
jurisdiction. 

Compliance involves actively identifying and understanding 
the relevant regulations, establishing internal controls and 
processes to meet those requirements, and consistently 
monitoring and reviewing operations to ensure ongoing 
adherence. It encompasses various aspects, such as legal, 
financial, operational, and ethical considerations, and aims to 
ensure that organizations operate within the boundaries of the 
law, maintain ethical standards, and fulfill their responsibilities 
to stakeholders, customers, and the public. 


The theory of regulatory compliance provides a framework for 
understanding the underlying principles and concepts that guide 
the compliance process. It encompasses several key elements 
that shape the approach to achieving and maintaining 
compliance. Here is an overview of the theory of regulatory 
compliance: 


Legal and Regulatory Environment: The theory recognizes that 

regulatory compliance is rooted in the legal and regulatory 

landscape. It acknowledges the importance of identifying and 
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understanding applicable laws, regulations, and standards that 
govern an industry or jurisdiction. 


Risk Management: The theory emphasizes the proactive 
identification, assessment, and management of risks associated 
with non-compliance. It highlights the need to establish robust 
risk management processes to mitigate legal, financial, 
operational, and reputational risks. 


Policies and Procedures: Effective compliance requires the 
development and implementation of comprehensive policies 
and procedures. The theory underscores the significance of 
clear, well-documented, and communicated policies that guide 
employees in adhering to regulatory requirements. 


Internal Controls: The theory emphasizes the establishment of 
internal controls to ensure compliance. This involves designing 
and implementing systems, processes, and checks that monitor 
and mitigate risks, detect and prevent non-compliance, and 
promote accountability. 


Training and Awareness: Recognizing the role of individuals in 
compliance, the theory highlights the importance of training 
programs and awareness initiatives. It emphasizes educating 
employees about applicable regulations, ethical standards, and 
the organization's compliance obligations. 


Monitoring and Auditing: The theory acknowledges the need 
for ongoing monitoring and auditing to assess compliance 
effectiveness. Regular internal audits, reviews, and assessments 
help identify gaps, weaknesses, and areas for improvement, 
ensuring continuous compliance efforts. 
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Reporting and Documentation: The theory stresses the 
significance of accurate and timely reporting of compliance 
activities. It underscores the need to maintain proper 
documentation, records, and evidence of compliance processes, 
actions taken, and outcomes achieved. 


Compliance Culture: The theory recognizes that compliance is 
not solely a set of rules and processes but also a cultural 
mindset. It highlights the importance of fostering a culture of 
compliance within an organization, where integrity, ethics, and 
adherence to regulations are valued and embedded in the 
organizational DNA. 


Accountability and Enforcement: The theory acknowledges 
that compliance requires accountability for non-compliance. It 
recognizes the role of regulatory bodies, internal enforcement 
mechanisms, and disciplinary actions in promoting compliance 
and deterring violations. 


Continuous Improvement: Finally, the theory emphasizes the 
need for continuous improvement in compliance efforts. It 
encourages organizations to learn from past experiences, adapt 
to evolving regulations, embrace emerging best practices, and 
strive for excellence in their compliance initiatives. 


By understanding and applying the theory of regulatory 
compliance, organizations can establish a solid foundation for 
effective compliance management, minimize risks, and uphold 
legal and ethical standards in their operations. 
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Ensuring Legal and Ethical Practices 


Compliance with laws and regulations is a fundamental aspect 
of the theory of regulatory compliance. It recognizes that 
adherence to legal and regulatory requirements is crucial for 
organizations to operate within the boundaries set by governing 
bodies and to fulfill their obligations to stakeholders. Here are 
key points related to compliance with laws and regulations in 
the context of the theory of regulatory compliance: 


Understanding Applicable Laws: The theory emphasizes the 
importance of identifying and comprehending the specific laws 
and regulations that pertain to an organization's industry, 
jurisdiction, and operational activities. This involves staying 
updated with changes in regulations and interpreting their 
implications for the organization. 


Regulatory Research and Analysis: Organizations need to 
conduct thorough research and analysis to determine how laws 
and regulations apply to their operations. This includes 
examining regulatory frameworks, guidance documents, legal 
precedents, and industry-specific requirements. 


Compliance Obligations: The theory recognizes _ that 
compliance obligations vary based on the nature of the 
organization's activities. It stresses the need to determine the 
specific requirements, obligations, and standards that the 
organization must meet to ensure legal and regulatory 
compliance. 


Compliance Program Development: To achieve compliance, 
the theory highlights the importance of developing a 
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comprehensive compliance program tailored to the 
organization's needs. This involves establishing policies, 
procedures, and controls that align with legal and regulatory 
requirements. 


Regulatory Reporting and Filings: Compliance entails fulfilling 
reporting obligations to regulatory authorities. The theory 
emphasizes the significance of timely and accurate reporting, 
including financial statements, disclosures, permits, licenses, 
certifications, and other regulatory filings. 


Compliance Monitoring and Auditing: The theory underscores 
the need for ongoing monitoring and auditing of compliance 
efforts. Regular reviews help identify potential compliance 
gaps, assess the effectiveness of controls, and ensure corrective 
actions are taken to address non-compliance. 


Compliance Documentation: Documentation plays a critical 
role in compliance. The theory highlights the importance of 
maintaining accurate and comprehensive records of compliance 
activities, including policies, procedures, training materials, 
audit reports, incident reports, and evidence of compliance. 


Compliance Risk Assessment: Organizations should conduct 
compliance risk assessments to identify and evaluate potential 
risks associated with non-compliance. This allows for the 
implementation of risk mitigation strategies, such as internal 
controls, training programs, and monitoring systems. 


Enforcement and Consequences: The theory acknowledges that 

non-compliance can lead to legal and financial consequences. 

It emphasizes the need for organizations to understand the 
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potential penalties, fines, sanctions, and reputational damage 
that can result from violations of laws and regulations. 


Regulatory Engagement and Communication: Organizations 
should actively engage with regulatory authorities and maintain 
open lines of communication. The theory emphasizes the 
importance of understanding regulatory expectations, seeking 
guidance when needed, and participating in _ industry 
consultations. 


By emphasizing compliance with laws and regulations, the 
theory of regulatory compliance aims to ensure that 
organizations operate within legal boundaries, mitigate risks, 
protect stakeholders, and maintain a strong ethical foundation 
in their operations. 


Protection of Consumers and Public Interest 


Protection of consumers and the public interest is a fundamental 
objective of regulatory compliance. Regulatory compliance 
refers to the adherence of individuals, organizations, or 
businesses to laws, regulations, and guidelines set forth by 
governing bodies or regulatory authorities. It aims to ensure that 
entities operate in a manner that safeguards the interests of 
consumers and the general public. 


The theory behind regulatory compliance is rooted in the belief 
that certain industries or activities require oversight and 
regulation to prevent harm, ensure fair competition, and 
maintain public trust. By establishing rules and standards, 
regulatory bodies seek to create a level playing field, promote 
transparency, and protect the well-being of consumers. 
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Key principles and considerations associated with regulatory 
compliance for the protection of consumers and public interest 
include: 


Consumer Protection: Regulatory compliance frameworks 
typically include provisions to safeguard consumers from 
fraudulent, deceptive, or unfair practices. This involves 
regulations related to product safety, labeling, advertising, 
pricing, warranties, and consumer rights. 


Public Health and Safety: Compliance regulations often address 
public health and safety concerns. For instance, in the 
pharmaceutical industry, compliance with drug safety 
regulations ensures that medications meet quality standards and 
do not pose unreasonable risks to patients. 


Market Integrity: Regulatory compliance helps maintain the 
integrity of markets by prohibiting anti-competitive behavior, 
ensuring fair trading practices, and preventing market 
manipulation or insider trading. These regulations promote fair 
competition and protect consumers from monopolistic 
practices. 


Data Protection and Privacy: With the increasing prevalence of 
data-driven technologies, regulatory compliance frameworks 
emphasize the protection of personal information and privacy 
rights. Regulations like the European Union's General Data 
Protection Regulation (GDPR) aim to safeguard consumer data 
and establish guidelines for its lawful collection, storage, and 
use. 
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Financial Stability: Regulatory compliance plays a crucial role 
in the financial sector to prevent fraud, money laundering, and 
unethical practices that can destabilize markets or harm 
consumers. Regulations impose standards for capital adequacy, 
risk management, disclosure, and consumer financial 
protection. 


Ethical Considerations: Compliance regulations often 
incorporate ethical considerations to ensure responsible and 
ethical behavior by individuals and organizations. This may 
involve guidelines on corporate governance, social 
responsibility, environmental sustainability, or labor practices. 


To ensure effective regulatory compliance, regulatory bodies 
conduct inspections, audits, and enforcement actions. Non- 
compliance can result in penalties, fines, or legal actions against 
the offending parties. Moreover, compliance management 
systems, internal controls, and self-regulatory mechanisms are 
employed by organizations to proactively adhere to regulatory 
requirements and promote a culture of compliance. 


Overall, the theory of regulatory compliance revolves around 
the idea that by setting and enforcing rules, regulators can 
protect consumers, preserve public interest, and maintain the 
stability and fairness of various sectors in society. 


Financial Stability and Risk Management 
Financial stability and risk management are critical components 
of regulatory compliance. The theory of regulatory compliance 


emphasizes the importance of establishing and enforcing 
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regulations to ensure the stability and integrity of financial 
systems, protect consumers, and mitigate systemic risks. 


Here are some key aspects of the theory of regulatory 
compliance related to financial stability and risk management: 


Prudential Regulation: Prudential regulation focuses on 
ensuring the soundness and stability of financial institutions, 
such as banks, insurance companies, and investment firms. 
Regulatory compliance frameworks impose requirements 
related to capital adequacy, risk management, liquidity, and 
asset quality to prevent excessive risk-taking and protect the 
financial system from disruptions. 


Systemic Risk Mitigation: Regulatory compliance measures 
aim to identify and mitigate systemic risks that can have 
widespread adverse effects on the financial system. This 
includes regulations on risk concentration, interconnectedness, 
and exposure limits to prevent the domino effect of failures and 
contagion across institutions. 


Risk Assessment and Monitoring: Regulatory compliance 
frameworks often require financial institutions to conduct 
thorough risk assessments and implement robust risk 
management practices. This involves identifying, measuring, 
and monitoring various types of risks, including credit risk, 
market risk, liquidity risk, and operational risk. Compliance 
regulations may prescribe specific methodologies, reporting 
requirements, and stress testing to ensure that risks are 
adequately identified and managed. 
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Transparency and Disclosure: Regulatory compliance promotes 
transparency in financial markets by requiring financial 
institutions to provide accurate and timely disclosure of 
relevant information to investors, regulators, and the public. 
This includes financial reporting, disclosures of risk exposures, 
and information about the institution's financial health. 
Transparent reporting helps stakeholders make informed 
decisions, enhances market efficiency, and fosters trust in the 
financial system. 


Consumer Financial Protection: Regulatory compliance 
frameworks incorporate measures to protect consumers in 
financial transactions. This includes regulations on fair lending 
practices, disclosure requirements for financial products and 
services, and regulations against abusive or predatory practices. 
These regulations aim to ensure that consumers are treated 
fairly, have access to transparent information, and are protected 
from fraudulent or deceptive practices. 


Regulatory Oversight and Enforcement: Regulatory 
compliance is reinforced by regulatory bodies that oversee 
financial institutions, enforce compliance, and impose penalties 
for non-compliance. These regulatory authorities monitor 
institutions' compliance with regulations, conduct audits and 
examinations, and take enforcement actions when violations are 
identified. Such oversight ensures accountability and promotes 
a culture of compliance within the financial industry. 


By adhering to regulatory compliance requirements, financial 

institutions are expected to minimize risks, enhance stability, 

and maintain the confidence of investors and the public. 

Compliance management systems, internal controls, and risk 
160 


management frameworks are utilized by financial institutions to 
meet regulatory obligations and proactively manage risks. 


Overall, the theory of regulatory compliance underscores the 
role of regulations in promoting financial stability, mitigating 
risks, protecting consumers, and maintaining the integrity of 
financial systems. Compliance with these regulations helps 
build a resilient financial sector that can withstand shocks and 
contribute to overall economic stability. 


Preserving Competitive Market Environment 


Preserving a competitive market environment is essential for 
fostering innovation, encouraging efficiency, and benefiting 
consumers. The theory of regulatory compliance is closely 
linked to this objective, as it involves establishing and enforcing 
rules and regulations that promote fair competition and prevent 
anti-competitive practices. 


The theory of regulatory compliance is based on the idea that 
regulatory frameworks can help create a level playing field for 
all market participants. By setting clear rules and standards, 
regulators aim to ensure that businesses operate within the 
bounds of fair competition. Compliance with these regulations 
helps prevent monopolistic behavior, collusion, price-fixing, 
and other practices that could harm competition. 


Here are a few key principles related to preserving a 


competitive market environment and the theory of regulatory 
compliance: 


161 


Anti-Trust Laws: Anti-trust laws are designed to promote 
competition by preventing the abuse of market power. They 
prohibit practices such as monopolies, cartels, price-fixing, and 
mergers that may substantially lessen competition. Regulators 
enforce these laws to preserve a competitive landscape and 
protect consumer interests. 


Market Entry and Exit: Regulatory frameworks should facilitate 
the entry of new businesses into the market while allowing 
existing ones to exit if they are unable to compete effectively. 
Barriers to entry, such as excessive licensing requirements or 
unfair regulations, can hinder competition. Regulatory 
compliance should aim to reduce these barriers and ensure fair 
access for all participants. 


Consumer Protection: A competitive market environment 
should prioritize consumer welfare. Regulatory compliance 
plays a crucial role in safeguarding consumer interests by 
ensuring transparency, fair pricing, quality standards, and 
adequate information disclosure. Consumer protection laws and 
regulations address issues such as misleading advertising, 
product safety, and fair dispute resolution mechanisms. 


Enforcement and Monitoring: Regulatory agencies are 
responsible for enforcing compliance with regulations. They 
monitor market activities, investigate potential violations, and 
take appropriate enforcement actions when necessary. Effective 
enforcement requires sufficient resources, expertise, and 
collaboration among regulators, ensuring a level playing field 
for all participants. 
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International Cooperation: In a globalized economy, preserving 
a competitive market environment requires international 
cooperation. Collaboration between regulatory authorities 
across jurisdictions can help address cross-border anti- 
competitive practices, harmonize regulatory standards, and 
promote fair competition in the global marketplace. 


Overall, the theory of regulatory compliance supports the 
notion that well-designed and effectively enforced regulations 
can foster a competitive market environment. By promoting fair 
competition, preventing anti-competitive practices, and 
protecting consumer interests, regulatory compliance 
contributes to a healthy and vibrant marketplace. 


Establishing Trust and Credibility 


Establishing trust and credibility is crucial for regulatory 
compliance efforts. The theory of regulatory compliance 
recognizes that trust is essential in fostering cooperation 
between regulatory authorities, businesses, and _ other 
stakeholders. Trust is built when regulations are transparent, 
consistently enforced, and perceived as fair and unbiased. 


Here are some key aspects of establishing trust and credibility 
in the context of regulatory compliance: 


Transparency: Transparency is a fundamental principle in 
regulatory compliance. Regulations and their enforcement 
processes should be clearly communicated and accessible to all 
stakeholders. Openness helps build trust by ensuring that the 
rules are known and understood by businesses and individuals, 
reducing uncertainty and promoting voluntary compliance. 
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Consistency: Consistency in applying regulations is critical for 
building trust. Regulators should strive to enforce regulations 
uniformly and without favoritism or discrimination. Consistent 
enforcement establishes a level playing field, fostering trust 
among market participants who know that everyone is subject 
to the same rules. 


Accountability: Regulatory authorities should be accountable 
for their actions. This includes being transparent about 
decision-making processes, justifying regulatory actions, and 
providing avenues for recourse and appeal. Accountability 
mechanisms help prevent abuse of regulatory power and build 
trust by demonstrating fairness and impartiality. 


Collaboration and Engagement: Regulatory compliance efforts 
benefit from collaboration and engagement with various 
stakeholders. This includes businesses, industry associations, 
consumer groups, and experts. Involving stakeholders in the 
regulatory process helps ensure that regulations are practical, 
effective, and well-understood. Collaboration also enhances 
trust by incorporating diverse perspectives and building 
consensus. 


Risk-Based Approach: A risk-based approach to regulation can 
contribute to trust and credibility. It involves assessing risks, 
prioritizing enforcement efforts based on the potential harm to 
the public or the market, and proportionately allocating 
regulatory resources. This approach demonstrates _ that 
regulatory actions are driven by objective evaluations and the 
need to address significant risks, enhancing trust in the 
regulatory system. 
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Continuous Improvement: Regulatory compliance should be a 
dynamic and evolving process. Regular evaluation and 
improvement of regulations and enforcement mechanisms are 
essential for maintaining trust and credibility. Regulators 
should engage in periodic reviews, solicit feedback from 
stakeholders, and adapt regulations to changing market 
dynamics and emerging challenges. 


Effective Communication: Clear and effective communication 
is vital for establishing trust. Regulators should communicate 
expectations, obligations, and changes in regulations in a timely 
and accessible manner. Communication channels should be 
open to addressing queries, providing guidance, and clarifying 
regulatory requirements, fostering trust by ensuring 
transparency and promoting compliance. 


In summary, trust and credibility are foundational elements of 
successful regulatory compliance. By promoting transparency, 
consistency, accountability, collaboration, and _ effective 
communication, regulatory authorities can establish a trusted 
regulatory framework that fosters compliance and cooperation 
among stakeholders. 


Penalties and Consequences of Non-Compliance 


Regulatory compliance refers to the act of adhering to laws, 

regulations, guidelines, and standards set forth by governing 

bodies or regulatory agencies. Non-compliance occurs when 

individuals, organizations, or businesses fail to meet these 

requirements. The penalties and consequences of non- 

compliance can vary depending on the specific regulations and 
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jurisdictions involved. Here are some common penalties and 
consequences: 


Fines and Monetary Penalties: Regulatory agencies often have 
the authority to impose fines and monetary penalties for non- 
compliance. The amount of the penalty may vary depending on 
the severity of the violation and the regulatory framework in 
place. These fines can be substantial and can significantly 
impact the finances of non-compliant entities. 


Legal Proceedings and Lawsuits: Non-compliance may lead to 
legal action, including lawsuits filed by affected parties or 
regulatory bodies. This can result in costly litigation, potential 
damages, and a tarnished reputation. 


License Revocation or Suspension: Certain industries and 
professions require licenses or permits to operate legally. Non- 
compliance can lead to the revocation or suspension of these 
licenses, effectively shutting down the business or preventing 
individuals from practicing their profession. 


Regulatory Audits and Inspections: Regulatory agencies may 
conduct audits and inspections to assess compliance. Non- 
compliant entities may face increased scrutiny, additional 
audits, or more frequent inspections, leading to disruption of 
operations and additional costs. 


Reputational Damage: Non-compliance can harm an 
organization's reputation, leading to loss of customer trust, 
decreased sales, and difficulty attracting new customers. 
Negative publicity and media attention can have long-lasting 
effects on brand value and perception. 
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Corrective Actions and Remediation Costs: In many cases, non- 
compliant entities are required to take corrective actions to 
address the violations. This may involve implementing new 
policies, procedures, or systems, as well as investing in training 
and education. The costs associated with these remediation 
efforts can be significant. 


Criminal Charges and Penalties: In cases of serious non- 
compliance, intentional violations, or fraudulent activities, 
criminal charges may be pursued. This can result in fines, 
imprisonment, or both, depending on the severity of the offense. 


The theory of regulatory compliance seeks to understand why 
individuals or organizations choose to comply or not comply 
with regulations. Factors influencing compliance behavior 
include perceived legitimacy of regulations, trust in regulatory 
agencies, the presence of effective enforcement mechanisms, 
and the perceived costs and benefits of compliance. The theory 
emphasizes the importance of clear communication, consistent 
enforcement, and proportionate penalties to achieve higher 
compliance rates. 


Compliance Programs and Frameworks 


Compliance programs and frameworks are designed to help 
organizations establish and maintain a culture of regulatory 
compliance. They provide a_ structured approach to 
understanding and meeting regulatory requirements, mitigating 
risks, and promoting ethical behavior. Additionally, 
compliance programs help organizations detect and address 
non-compliance issues promptly and effectively. 
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Here are some common compliance programs and frameworks: 


Compliance Management System (CMS): A CMS is a 
comprehensive framework that encompasses policies, 
procedures, processes, and controls to manage compliance 
within an organization. It includes elements such as risk 
assessment, compliance training, monitoring and auditing, 
incident reporting, and corrective action planning. 


ISO 19600: This international standard provides guidelines for 
establishing, implementing, evaluating, and improving a 
compliance management system. It emphasizes a risk-based 
approach to compliance and provides a framework for 
organizations to identify, analyze, and address their compliance 
obligations effectively. 


COSO Framework: The Committee of Sponsoring 
Organizations of the Treadway Commission (COSO) 
developed a framework that focuses on internal controls and 
risk management. While not specifically geared towards 
compliance, it provides a solid foundation for managing 
compliance risks within an organization. 


Federal Sentencing Guidelines (FSG): The U.S. Federal 
Sentencing Guidelines provide guidance for organizations on 
establishing effective compliance programs. They outline 
specific factors that organizations should consider when 
developing compliance programs, such as conducting risk 
assessments, implementing training and communication 
programs, and monitoring compliance. 
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Principle-Based Approach: The principle-based approach to 
compliance focuses on establishing a set of core principles and 
values that guide an organization's compliance efforts. It 
emphasizes ethical conduct, integrity, and accountability as the 
foundation for compliance programs. This approach encourages 
employees to make ethical decisions and act in accordance with 
the organization's values. 


The theory of regulatory compliance explores the factors that 
influence compliance behavior and the effectiveness of 
compliance programs. It recognizes that compliance is not 
solely driven by the fear of penalties but also by factors such as 
organizational culture, perceived legitimacy of regulations, and 
the presence of strong internal controls. The theory suggests 
that effective compliance programs should: 


Clearly communicate regulatory requirements and expectations 
to employees and stakeholders. 


Foster a culture of compliance by promoting ethical behavior, 
accountability, and integrity. 


Provide training and education to employees to enhance their 
understanding of compliance obligations. 


Implement monitoring and auditing mechanisms to detect and 
address non-compliance promptly. 


Establish strong internal controls and risk management 
processes to mitigate compliance risks. 
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Encourage reporting of potential compliance issues and provide 
channels for anonymous reporting. 


Continuously evaluate and improve the compliance program 
based on feedback and changes in regulations. 


By understanding the theory of regulatory compliance and 
implementing effective compliance programs, organizations 
can enhance their ability to meet regulatory requirements, 
manage risks, and uphold ethical standards. 


Role of Technology in Regulatory Compliance and monitoring 
and reporting tools 


Technology plays a crucial role in regulatory compliance by 
providing tools and systems that help organizations monitor and 
report their adherence to regulatory requirements. Here are 
some key ways technology supports regulatory compliance: 


Automation and Workflow Management: Technology enables 
the automation of various compliance processes, such as data 
collection, analysis, and reporting. Workflow management 
systems help streamline compliance tasks by providing clear 
processes and guidelines, ensuring consistent and efficient 
execution. 


Data Management and Analysis: Compliance often involves 
handling large volumes of data. Technology solutions, such as 
data management systems and analytics tools, facilitate the 
collection, storage, organization, and analysis of data for 
compliance purposes. These systems can identify patterns, 
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anomalies, and trends in the data, helping organizations detect 
and address compliance risks. 


Monitoring and Surveillance: Technology enables real-time 
monitoring and surveillance of activities, transactions, and 
communications to identify potential compliance violations. 
Advanced monitoring tools use algorithms and machine 
learning techniques to detect suspicious behavior, fraud, market 
manipulation, or any non-compliant activities. 


Reporting and Documentation: Compliance requires accurate 
and timely reporting to regulatory authorities. Technology 
offers reporting tools that help automate the creation of 
regulatory reports, ensuring the required information is 
captured, organized, and submitted in the appropriate format. 
These tools often include templates, data mapping capabilities, 
and integration with existing systems. 


Audit Trail and Documentation Management: Technology 
allows organizations to maintain a comprehensive audit trail 
and documentation of compliance activities. Digital systems 
enable the secure storage, retrieval, and tracking of compliance- 
related documents, making it easier to demonstrate compliance 
during audits or investigations. 


Risk Assessment and Compliance Monitoring: Technology 
supports risk assessment processes by providing tools for 
identifying, assessing, and prioritizing compliance risks. 
Compliance monitoring tools can continuously track regulatory 
changes and updates, ensuring organizations stay informed and 
adapt their compliance programs accordingly. 
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Training and Education: Technology can be utilized to deliver 
compliance training and educational materials to employees 
and stakeholders. Online learning platforms, webinars, and 
interactive modules can provide accessible and engaging 
compliance training programs, ensuring widespread 
understanding of regulatory requirements and promoting a 
culture of compliance. 


Overall, technology plays a vital role in enhancing the 
efficiency, accuracy, and effectiveness of regulatory 
compliance efforts. By leveraging technology, organizations 
can better manage compliance requirements, mitigate risks, and 
ensure adherence to regulations in an increasingly complex 
regulatory landscape. 


Conclusion Recap of the importance of the theory of regulatory 
compliance 


The theory of regulatory compliance is of great importance in 
various domains, particularly in legal and business contexts. It 
refers to the set of rules, regulations, and standards that 
individuals, organizations, and industries must follow to ensure 
compliance with applicable laws and regulations. 


Here are some key points highlighting the importance of the 
theory of regulatory compliance: 


Legal Compliance: Regulatory compliance ensures that 
individuals and organizations adhere to laws and regulations set 
forth by governing bodies. This helps maintain law and order in 
society and promotes fairness, transparency, and accountability. 
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Risk Mitigation: Compliance measures help identify and 
mitigate potential risks associated with non-compliance. By 
following regulations, organizations can minimize legal, 
financial, reputational, and operational risks. Compliance 
frameworks often include risk assessment and management 
components, enabling proactive risk mitigation. 


Consumer Protection: Compliance regulations often aim to 
protect consumers’ rights and interests. Compliance with 
consumer protection laws ensures fair business practices, 
prevents fraud, and enhances consumer trust in products and 
services. 


Data Privacy and Security: In the digital age, data privacy and 
security have become crucial concerns. Regulatory compliance 
frameworks, such as the General Data Protection Regulation 
(GDPR), enforce strict guidelines for handling personal data. 
Compliance helps safeguard sensitive information, maintain 
privacy, and prevent data breaches. 


Ethical Standards: Compliance extends beyond legal 
obligations and encompasses ethical standards. It encourages 
organizations to adopt ethical business practices, such as fair 
competition, anti-corruption measures, and environmental 
sustainability. Compliance frameworks often incorporate 
ethical guidelines to promote responsible conduct. 


Industry Standards: Many industries have specific regulatory 
compliance requirements tailored to their unique characteristics 
and risks. Compliance with industry-specific regulations 
ensures safety, quality, and standardization within the sector. 
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Examples include regulations in healthcare, finance, energy, 
and manufacturing. 


Reputation and Trust: Compliance with regulations builds a 
positive reputation for individuals and organizations. It 
demonstrates commitment to legal and ethical standards, 
fostering trust among customers, investors, and other 
stakeholders. A strong reputation for compliance can lead to 
increased business opportunities and competitive advantage. 


Legal Consequences: Non-compliance with regulatory 
requirements can have severe legal consequences, including 
fines, penalties, sanctions, and legal liabilities. Violations can 
result in damaged reputation, loss of business licenses, and even 
criminal charges. Compliance helps organizations avoid legal 
pitfalls and maintain a good standing with regulatory 
authorities. 


Global Business Landscape: With increasing globalization, 
organizations often need to navigate complex regulatory 
frameworks across multiple jurisdictions. Understanding and 
complying with international regulations is essential for 
expanding businesses, facilitating international trade, and 
avoiding legal disputes. 


Continuous Improvement: The theory of regulatory compliance 
emphasizes the need for continuous improvement. Compliance 
programs encourage regular monitoring, self-assessment, and 
adaptability to evolving regulations. This fosters a culture of 
compliance and enables organizations to stay up to date with 
changing legal requirements. 
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In summary, the theory of regulatory compliance plays a vital 
role in promoting legality, ethical conduct, risk management, 
and trust in various domains. It ensures adherence to laws, 
protects consumers, mitigates risks, and helps organizations 
thrive in a complex regulatory landscape. 


Here are additional more recent posts from RIKINotes that I 
wanted to include in this handbook because they really help 
to round out the discussion on regulatory compliance, 
licensing measurement and monitoring systems and keep it 
current, 


When it comes to licensing measurement and monitoring 
systems, risk assessment is the driving force in making 
licensing decisions, remembering the mantra: “Do No Harm“. 
There have been several posts giving examples in how one does 
this with risk assessment and key indicator methodologies 
which are the predominant approaches to differential 
monitoring. These methodologies are derived by two very 
different mathematical models, one based upon Likert scaling 
and weighting; the other based on predictive scaling and 
regulatory compliance history. However, what they have in 
common is a basic risk aversion. 


With risk assessment rules, the selection process via a 
weighting methodology is critical in selecting those rules that 
place individuals at greatest risk of harm, and then making 
certain that these rules are always in regulatory compliance. 
With predictive rules, the selection process is through 
regulatory compliance history in general as well as with each 
individual rule. The key here is to make certain that the effect 
size is sufficiently large so that there are no false negatives. 


175 


The licensing decision process needs to ensure at all times that 
there is no regulatory non-compliance with the risk assessment 
rules and that there are no false negatives where general 
regulatory non-compliance is found with some other rule when 
the predictive rules are all in-compliance. In order to have an 
effective and efficient differential monitoring approach both 
these conditions must be met for the licensing system to work 
as it is intended with abbreviated inspection reviews. It is only 
by having this in place will a licensing agency feel confident 
that the necessary risk mitigation has been implemented in 
making licensing decisions. 


The next two posts are intimately tied together and should be 
read in close proximity of each other to understand the 
methodologies presented. 


Risk Assessment and Key Indicator methodologies are two 
approaches utilized in differential monitoring systems for 
generating an abbreviated inspection by only looking at a core 
set of rules based upon statistical predictor or risk assessment 
algorithms. In this post the matrix (pictured below) utilized to 
generate these core sets of rules are depicted and with a matrix 
that determines their respective validation status based upon 
subsequent studies. 


The first matrix display (KIM Matrix) deals with the Key 
Indicator Methodology (KIM) and demonstrates how key 
indicator rules are determined by measuring each potential rule 
and comparing it to the regulatory compliance history for the 
respective set of all rules for a given jurisdiction in which the 
programs are grouped into either a high (Full or substantial 
regulatory compliance with all rules) or low compliant groups 
(several or more violations of rules). From the matrix, it is clear 
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that for a rule to become an indicator rule, there needs to be a 
very high correlation between the rule being in compliance with 
the high group and out of compliance with the low group. It is 
only when this occurs that the rule will distinguish between 
high and low compliance and be a predictor rule. The other two 
cells should occur less frequently but there will be some 
occurrences when these do occur and when they do, these rules 
will not make the threshold of becoming indicator rules. So Key 
Indicator Predictor Rules increase performance by predicting 
overall regulatory compliance. 


The second matrix display (RAM Matrix) deals with the Risk 
Assessment Methodology (RAM) and demonstrates how risk 
assessment rules are determined by measuring each potential 
rule by the amount of risk of morbidity or mortality a client is 
placed in because of non-compliance with the specific rule and 
how likely will this occur. As one can see, the cell which 
contains high risk rules, and they are likely to occur would be 
included on the risk assessment tool. All the other cells are color 
coded in decreasing risk and likelihood categories and a 
jurisdiction can determine the appropriate thresholds. More risk 
rules would be included for a risk aversive approach while less 
risk rules would be included for a more lenient approach or 
because the number of key indicator rules are sufficient to 
ensure the health and safety of the clients being served. So, Risk 
Assessment Rules decrease risk to clients but are not predictive 
rules of overall regulatory compliance. 


The last matrix display (KIM/RAM Validation Matrix) is used 

after the KIM and RAM tools are actually used to validate that 

they are working as intended. KIM should be statistically 

predicting overall compliance with all the rules (Rules in 
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Compliance cell), while RAM should be mitigating risk in the 
program by always having the high-risk rules in compliance 
(also Rules in Compliance cell). Part of the KIM validation 
strategy is that the opposite should also occur in that when the 
KIM tool has indicator rules out of compliance, it should 
statistically predict rules out of compliance with other rules (the 
Rules Out of Compliance cell). Something that can occur but 
needs to be eliminated are the false negatives in which the KIM 
is in compliance but there is non-compliance detected 
elsewhere in the rules. When full compliance is used for the 
high compliant group in the KIM Matrix, this eliminates this 
from happening. But if substantial compliance is used as the 
criterion for the high compliant group, then this can become 
problematic. If substantial compliance is used as the threshold 
for the high compliant group, a multiplier needs to be applied 
to rule out the likelihood of false negatives (please see the blog 
post on this algorithm adjustment posted back in January of this 
year or look at the description provided below the matrices). 
False positives are possible also but are not of overall concern 
from a safety point of view but are a concern from a 
psychometric standpoint and additional research needs to be 
done to determine the cause. 
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KIM Generator 


Rule In 
Compliance 


Rule Out of 
Compliance 


High Risk/High 
Likely 


Med Risk/High 
Likely 


Low Risk/High 
Likely 


KIM Matrix 


High Compliant 
Group 


RAM Matrix 
High Risk/Med 
Likely 


Med Risk/Med 
Likely 


Low Risk/Med 
Likely 


Low Compliant 
Group 


High Risk/Low 
Likely 


Med Risk/Low 
Likely 


Low Risk/Low 
Likely 


KIM/RAM Validation Matrix 


KIM/RAM 
Validator 


KIM/RAM In 


Compliance 


KIM/RAM Out 
Compliance 


Rules In 
Compliance 


Yes/Yes: OK 
KIM/RAM 


No/Yes: False 
Positive 
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Rules Out of 
Compliance 


Yes/No: False 
Negative 


No/No: OK KIM 


Hopefully this post helps licensing administrators, licensing 
researchers, and regulatory scientists to see the logic behind the 
differential monitoring methodologies of key indicator and risk 
assessment and how best to take advantage of both. 


There are two other blog posts on the risk assessment (RAM) 
and key indicator (KIM) matrices posted last year (2022) and 
the year before (2021) demonstrating differences and 
similarities. In this post, there is an attempt to build upon the 
previous posts and to enhance some of these differences and 
similarities. Let’s start with a narrative description followed by 
a chart/matrix comparison. 


Risk Assessment (RAM) is generally depicted as a 3 x 3 matrix 
(pictured below) with risk on one axis and prevalence on the 
other axis; while Key Indicators (KIM) is generally depicted as 
a 2 x 2 matrix in which one axis measures individual rule 
compliance and the other axis measures overall regulatory 
compliance or compliance history. RAM deals with individual 
rules with a weight while KIM deals with aggregate rules and 
high and low regulatory compliance. RAM rules are heavily 
weighted while KIM rules are medium weighted. RAM is 
hardly ever out of compliance while KIM has a good deal of 
non-compliance to distinguish the high compliant group from 
the low compliant group. RAM uses likert scale and means; 
KIM uses correlational analyses and prediction. RAM is expert 
opinion while KIM is data driven 
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RAM/KIM Matrix: Risk Assessment and Key Indicators: 3x3 Matrix 
Demonstrating Relationships between KIM and RAM 


High Risk/High High Risk/Med High Risk/Low 
Prevalence Prevalence Prevalence 
Med Risk/High Med Risk/Med Med Risk/Low 
Prevalence Prevalence Prevalence 
Low Risk/High Low Risk/Med Low Risk/Low 
Prevalence Prevalence Prevalence 


In the above 3 x 3 Matrix: Risk x Prevalence are listed across 
the axis, in which RAM is preventing high risk, high prevalence 
but in reality RAM rules are very low prevalence, low non- 
compliance. KIM rules are usually med risk and prevalence. 


The above matrix and narrative provides additional 
enhancements to the differences and similarities between risk 
assessment and key indicator rules. As one can see, there are 
some basic differences but at the same time there is a deep 
common structure that underlies both. These are important 
attributes to consider before using these _ statistical 
methodologies as part of a differential monitoring approach. 
But the bottom line when using either RAM or KIM, or 
RAM+KIM, all RAM and KIM rules must be in compliance at 
all times. Remember it is not about more or less rules in total, it 
is about compliance with the right rules. 


Let’s take this to the next step and think about this more broadly 
and relate it to the larger research literature dealing with 
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businesses. Risk assessment and key performance indicators 
(KPIs) are two important concepts in business management. 
Risk assessment is the process of identifying, evaluating, and 
managing risks to an organization’s objectives. KPIs are 
metrics that measure an organization’s performance against its 
objectives. 


The two concepts are related in that risk assessment can help 
organizations identify and prioritize risks that could impact 
their KPIs. For example, if an organization’s KPI is to increase 
sales by 10%, then risk assessment can help the organization 
identify risks that could prevent it from achieving this goal, 
such as a competitor launching a new product or a change in 
customer behavior. 


Once risks have been identified, organizations can develop 
mitigation strategies to reduce the likelihood or impact of those 
risks. KPIs can be used to track the effectiveness of these 
mitigation strategies. For example, if an organization is 
concerned about a competitor launching a new product, it could 
track its sales data to see if there has been a decrease in sales 
since the competitor launched its product. 


By integrating risk assessment and KPIs, organizations can 
improve their ability to identify, manage, and mitigate risks to 
their objectives. This can help organizations achieve their goals 
and objectives more effectively. 


Here are some examples of how risk assessment and KPIs can 
be used together: 
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A bank might use risk assessment to identify the risks 
of fraud and theft. The bank could then use KPIs to track 
the number of fraudulent transactions and the amount of 
money lost to fraud. This information could be used to 
develop mitigation strategies, such as implementing 
new security measures or training employees on how to 
spot and prevent fraud. 

A manufacturing company might use risk assessment to 
identify the risks of product recalls and safety incidents. 
The company could then use KPIs to track the number 
of product recalls and the number of safety incidents. 
This information could be used to develop mitigation 
strategies, such as improving product quality or 
implementing new safety procedures. 

A retail company might use risk assessment to identify 
the risks of natural disasters and supply chain 
disruptions. The company could then use KPIs to track 
the number of natural disasters that occur in its region 
and the number of supply chain disruptions that occur. 
This information could be used to develop mitigation 
strategies, such as developing contingency plans or 
building up inventory. 


By integrating risk assessment and KPIs, organizations can 
improve their ability to identify, manage, and mitigate risks to 
their objectives. This can help organizations achieve their goals 
and objectives more effectively. 


I want to continue the discussion related to the relationship 
between risk assessment and key performance indicators. [have 
posted about this relationship and other assorted concepts and 
ideas related to it in several previous blog posts I posted earlier 
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this year. In this post I would like to see if I can tie some of 
these ideas and concepts together and show how risk and 
performance are more closely related and how to take 
advantage of this relationship. 


These ideas percolated from a conversation and discussions I 
have been having with a colleague about a webinar we will be 
doing together where he suggested the use of a graphic to help 
to explain the essence of key performance indicators. His 
graphic was to be an airplane cockpit and all the gauges present 
on the dashboard that a pilot is looking at. A great deal of data 
and information to process but s/he focused on about 5-6 gauges 
that were the most important in flying the plane and really told 
the pilot if things were ok or not and when s/he needed to check 
the other gauges because these key performance indicator and 
risk assessment gauges were telling s/he something was not 
quite right. I would guess that two of these gauges were the 
altimeter and speed gauges which I would include as risk 
assessment gauges and a third gauge would have been the fuel 
gauge which I would include as a key performance indicator. 


Why did I break these gauges down into the two major areas of 
risk assessment and key performance? Here is my thinking: the 
altimeter tells the pilot how close to the ground and a potential 
crash and the speed helps to prevent a stall of the aircraft. Both 
are high risk factors and things we would want to mitigate. The 
fuel tank is important to know how much fuel the pilot has left; 
in, and of itself, not necessarily a risk factor unless it becomes 
too low but will impact performance because it determines how 
far the pilot can fly the plane. 
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A similar scenario could be played out with driving a car. Speed 
is the risk factor as it increases, while the gas tank gauge is the 
key performance indicator determining how far we can go and 
how much we are getting per gallon of gas which is an indicator 
on many newer models. 


Let’s try this out in a totally different industry and scenario, 
such as the pharmaceutical/drug industry. When finding out if 
a new drug will work or not, there is a delicate balance of risk- 
benefit or risk-performance. Same concept, just different 
terminology being used. For risk assessment, either not taking 
the drug or taking too much of the drug will not be in the best 
interest of the patient. Too little or not at all the patient dies 
because the disease progresses. If the patient takes too much of 
the drug, given the side effects, the patient dies. The key 
performance indicator or benefit is finding the right target 
dosage of the drug which effectively keeps the patient alive and 
gets better or at least not any worse. 


Another example, one that I share somewhat reluctantly 
because some people may take offense but I think it is an 
effective example, the Ten Commandments. I actually have 
posted this earlier in a blog post as an example if one is 
interested in looking at this in more detail (May 2022). With the 
Ten Commandments, think of “Thou Shalt not Kill” as a risk 
assessment rule and “Thou Shall not Steal” as a key 
performance indicator. Obviously the consequences of the first 
are much greater than the second where one is literally stealing 
someone’s life, which is the underlying structure of the 
relationship between risk assessment and key performance 
indicators. 
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So let’s delve into this relationship of performance and risk 
mitigation based upon the above examples and see how they are 
all tied together. Risk mitigation (Do No Harm) is sort of the 
book ends of the relationship, too much or too little is not a good 
thing, while key performance (Do Good) is somewhere in 
between balancing effectiveness with efficiency and finding the 
right balance of rules and recommended standards (The essence 
of the Theory of Regulatory Compliance). Remember I am 
addressing regulatory compliance data and not social science 
data in general although it would be interesting to see how this 
relationship of performance and risk assessment plays out in the 
larger context of the social sciences. I have a funny feeling that 
many relationships of social science variables are more 
nonlinear than linear in nature. 


How are risk assessment and key performance indicators 
determined? Risk assessment rules are generally determined by 
expert opinion and group consensus either using or not using a 
Likert type Scale (Stepping Stones to Caring for Our Children 
and Caring for Our Children Basics are examples). Key 
performance indicators are determined from actual data, 
generally regulatory compliance history utilizing a regulatory 
compliance statistical methodology that results in the rule’s 
predictive ability (the statistical methodology is highlighted on 
this website in the publications section as well as on the 
National Association for Regulatory Administration’s 
(NARA) website https://www.naralicensing.org/key- 
indicators)(ASPE’s Thirteen Quality Indicators and the Early 
Childhood Program Quality Indicators Scale are examples 
(see previous blog posts on all these)). From a licensing 
measurement perspective, risk assessment rules are generally 
always in regulatory compliance because the rules place clients 
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at such great risk; while key performance indicators do not 
place clients at high risk as with risk assessment rules, generally 
have some non-compliance, just enough to distinguish between 
the high performers and the mediocre performers. 


This relationship is made possible because of the regulatory 
compliance theory of diminishing returns/the ceiling effect 
between regulatory compliance and program quality where we 
are really forced to look for a paradigm shift when it comes to 
licensing and program monitoring. The “One Size Fits All” a 
very absolute approach needs to be replaced with a more 
relative approach, such as “Differential Monitoring” and once 
this paradigm shift is made it naturally leads us to identifying 
risk assessment rules and key performance indicator rules. It 
really changes our frame of reference in establishing a proper 
balance between regulatory compliance and program quality 
standards. 


To summarize, too few or too many rules are not a good 
outcome, it is finding the proper balance of the “right rules’”’, 
finding that balance between effectiveness and efficiency, 
between risk mitigation and optimun performance. Let me leave 
you with this statement as an algorithm where TRC = Theory 
of Regulatory Compliance; RA = Risk Assessment; KI = Key 
Performance Indicator; RC = Regulatory Compliance; and PQ 
= Program Quality: TRC =) (RA + KD) => (RC + PQ) 
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The Public Policy Implications of the Regulatory 
Compliance Theory of Diminishing Returns, Regulatory 
Compliance Scaling, and the Program Quality Scoring 
Matrix along with Integrative Monitoring 


This technical research note/abstract provides a data matrix 
(below table) depicting the relationship between regulatory 
compliance and program quality. The data clearly demonstrate 
the regulatory compliance theory of diminishing returns which 
depicts the ceiling or plateau effect in this relationship between 
regulatory compliance data and program quality data. It also 
shows the difficulty one will have in distinguishing program 
quality differences at the full and high regulatory compliance 
levels but the ease in distinguishing program quality between 
low regulatory compliance and high regulatory compliance 
levels. 

This abstract unifies several separately developed regulatory 
compliance metrics and concepts by combining them into a 
single technical research note. The Regulatory Compliance 
Theory of Diminishing Returns (2019), The Regulatory 
Compliance Scale (2022), Integrative Monitoring (2023), and 
the Ten Principles of Regulatory Compliance Measurement 
(2023) have all been presented separately (all these papers are 
available for the interested reader on SSRN 
(https://www.ssrn.com/index.cfm/en/) or the Journal of 
Regulatory Science _(https://regsci-ojs-tamu.tdl.org/regsci/)). 


This abstract shows how they are all related and their 
importance in moving forward with regulatory compliance 
measurement in the future. The four jurisdiction’s (US 
National, Southern State, Western State, Canada) final reports 
are available at https://www.naralicensing.org/key-indicators 
for the interested reader. 
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Relationship of Regulatory Compliance Scale and Program Quality 
in Four Jurisdictions Matrix 


Reg Comp 
Scale 


US 
National 


Southern 
State 


Western 
State 


Canada 


Full 


3.03 (75) 


3.40 (15) 


4.07 (82) 


37.4 (44) 


High 


3.13 (135) 


4.00 (20) 


4.28 (69) 


38.5 (33) 


Mid 


2.87 (143) 


3.16 (32) 


4.17 (163) 


29.1 (36) 


Low 


2.65 (28) 


2.38 (2) 


3.93 (71) 


Significan 
Legend: 
US National = CLASS-IS scores 
Southern State and Western State = ECERS-R scores 
Canada = Canadian Program Quality Tool scores 


p<.001 


p<.05 


p<.001 


One-way ANOVA was performed on the data in each jurisdiction. 


Regulatory Compliance Scale (Reg Comp Scale (RCS)): 


Full = 0 violations (100% regulatory compliance with all rules/regulations) 
High = 1-2 violations; Mid = 3-9 violations; Low = 10+ violations 


The number in parentheses is the number of programs assessed in each jurisdiction. 


QualityScores 


4.28 


407 4 
= 3.13 3.16 
; a 287 » 
25 
15 
1 
05 
1 2 3 


Regulatory Compliance Scale and Quality 


4.47 
3.93 
2:65 
| | | 
= 


Regulatory Compliance Scale 


Legend: 1 = Full; 2 = High; 3 = Mid; 4 = Low. 
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Blue = US National; Orange = Southern State; Gray = Western State. Canada was left off 
because of different scaling. 


The above data matrix display is important for the early care 
and education (ECE) field because it demonstrates the 
relationship between licensing via regulatory compliance data 
measurement and program quality scores via CLASS, ERS, and 
the Canadian Quality Tool. The CLASS and ERS are well 
grounded ECE program quality tools while the Canadian 
Quality Tool is a new addition to the field. 


The data displayed show that a ceiling or plateau effect (quality 
scores did not change significantly as was generally the case 
with lower levels of regulatory compliance) occurred in all four 
jurisdictions when the regulatory compliance levels or the 
absence of rule/regulatory violations were compared to 
program quality scores as one moves from high regulatory 
compliance to full regulatory compliance (0 violations or 100% 
regulatory compliance with all rules). From a public policy 
point of view, it would lead us to believe that licensing is not 
the best avenue to program quality and that another 
intervention, such as Quality Rating and Improvement Systems 
(QRIS), would be necessary to enhance quality programming. 
What regulatory compliance and licensing does do is prevent 
harm and keep children in healthy and safe environments 
(please go to https://rikinstitute.com for examples to support 
this claim). So, from a public policy point of view, licensing is 
accomplishing its goals. But don’t expect licensing to address 
quality programming. For that to occur, either we need to 
continue our present system of licensing and Quality Initiatives, 
such as QRIS, as an add on; or infuse quality into the rules and 
regulations which has been suggested via a new form program 


monitoring called: integrative monitoring. 
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There are some other takeaways from the above data matrix that 
are significant contributions to the regulatory compliance 
measurement research literature, such as, how skewed the data 
are. Focus more on the number of programs rather than their 
quality scores for each of the Regulatory Compliance Scale 
levels. You will notice that most programs in each of the 
jurisdictions are either in full or high regulatory compliance and 
that there are few programs at the low end of the regulatory 
compliance scale. There is an unusually very high percentage 
of programs at full compliance. This also contributes to a lack 
of variance in the upper end of the regulatory compliance scale 
which can be problematic as indicated in the previous paragraph 
in distinguishing between the quality levels of programs. 


The importance of these four studies and the summary matrix 
above is to provide a context in how licensing and regulatory 
compliance data should be used in making public policy 
decisions, for example: is it more effective and efficient to 
require high or substantial regulatory compliance than full 
regulatory compliance with all rules and regulations to be 
granted a full license to operate? It appears prudent to continue 
with the US emphasis on QRIS as an add on quality initiative, 
especially in states where rules/regulations are at a minimal 
level. In Canada their emphasis has been more in line with an 
integrative monitoring approach in which quality elements are 
built in or infused within the rules and regulations themselves. 
This approach appears to work in a similar fashion and is an 
effective public policy initiative. Either approach appears to be 
an effective modality to increasing program quality; but are 
both equally efficient. 
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The purpose of this last RIKINotes post is to point out the 
intersections, differences and similarities of integrative, 
differential/inferential and coordinated monitoring as used in 
the monitoring of human service programs. Program 
monitoring has changed over the years in that not only has it 
grown in the types of monitoring done, such as process, 
compliance, outcome monitoring, etc.; but also, in the 
functional aspects of monitoring as delineated with integrative, 
differential, and coordinated monitoring. Much has been 
written in the research literature about the types of monitoring 
but not as much regarding the functional aspects of monitoring 
probably because it is much newer and has grown with the 
various types of monitoring being used in different contexts. 


Coordinated monitoring deals with monitoring across similar 
service types, for example, in early care and education, 
monitoring would be done using similar standards in Head 
Start, child care, preschool, etc. This is an effective and efficient 
approach which has been demonstrated through the creation 
and dissemination of Caring for Our Children Basics as a core 
set of standards for all these various settings. The US Dept of 
Health and Human Services has advocated this particular 
approach. 


Differential monitoring focusing on the use of abbreviated or 
targeted inspections of programs that have a history of high 
regulatory compliance with specific rules or standards. It means 
spending more time and doing a more comprehensive review of 
those programs having difficulty complying with specific rules, 
these can be based upon risk assessment or predictive value of 
overall compliance. This is a very efficient approach which has 
been demonstrated to save time in monitoring reviews. Many 
192 


states in the USA and provinces in Canada use this approach. 
The US Office of Head Start has experimented with the 
approach. 


Instrument-based program monitoring utilizes instruments, 
tools, or checklists for recording all data when a review or 
inspection is completed. It is different from the case review or 
anecdotal type of record keeping. This approach started in the 
late 1970's, early 1980's when it was introduced by the 
Children's Services Monitoring Transfer Consortium, a 
federally funded research project consisting of California, 
Michigan, West Virginia, Pennsylvania and New York City. Its 
development occurred parallel with the development of 
differential monitoring but with particular emphasis on the 
metrics or measurement domain when it came to tool 
development. The Child Development Program Evaluation 
Scale was a major tool developed from this initiative. 


Integrative monitoring is a relatively new approach to 
monitoring in which the emphasis is on integrating regulatory 
compliance rules with quality programming standards. Note the 
emphasis is on the rules and standards and not on who gets 
applied to those rules and standards nor how they get applied. 
However, combining integrative monitoring with differential 
monitoring is an interesting research focus which could be a 
very effective and efficient approach in combining these two 
perspectives. In the past, licensing and quality programming 
have generally been in their own silos when it comes to program 
monitoring. Integrative monitoring removes them from these 
silos and suggests building a continuous metric that starts with 
the health and safety aspects of rules and adds in the quality 
pieces on top of the rules. Presently, quality initiatives, such as 
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Quality Rating and Improvement Systems, Accreditation, and 
Professional Development systems are examples of standards 
that could be used to build upon health and safety licensing 
rules. 


There appears to be interest in pursuing an integrative 
monitoring approach in several jurisdictions in the early care 
and education field but this interest extends beyond and has 
been suggested more broadly by a recent article published in the 
Journal of Regulatory Science by Freer & Fiene (2023). 
Regulatory compliance and quality programming: Constraints 
and opportunities for integration, Volume 11, Number 1, 1-10 
(Journal of Regulatory Science). The interested reader may 
want to take a look at the article, it does provide a unique model 
for pursuing integrative monitoring. Also, this eHandBook on 
Licensing Measurement and Monitoring Systems: Regulatory 
science applied to human services regulatory administration 
available at https://RIKInstitute.com. provides the basics of 
licensing measurement and program monitoring metrics. 


So where does all this lead to. Potentially to an expansion of 
the Regulatory Compliance Scale to a new proposed Licensing 
and Quality Scale. 


Previous RIKINotes posts have introduced the Regulatory 
Compliance Scale (RCS), in this post, based upon the latest 
regulatory science research, this RCS can be expanded to a 
more comprehensive and all-inclusive Licensing and Quality 
Scale (LQS) which will seven components related to licensing 
the program quality. 
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The seven components are the following: the Regulatory 
Compliance Scale, risk assessment rules, key indicator rules, 
quality indicator standards, complaints about the facility, key 
indicator criteria being satisfied, and overall regulatory 
compliance history. 


The Regulatory Compliance Scale (RCS-see table below) is a 
Likert type scale that has 1 —7 scaling where 7 = full regulatory 
compliance (no rule violations); 5 = substantial regulatory 
compliance (1-2 rule violations); 3 = moderate regulatory 
compliance (3-9 rule violations); and 1 = low regulatory 
compliance (10+ rule violations). The RCS is based upon 40 
years of research and the corresponding international regulatory 
compliance and quality databases. 


Regulatory Compliance Scale (RCS) 


RCS Level Violations 

Full 7 0 violations 
Substantial 5 1-2 violations 
Moderate 3 3-9 violations 

Low 1 10+ violations 


Risk Assessment Rules (RAR) are those rules which have been 
determined to place children at greatest risk for 
mortality/morbidity. These identified rules are generally always 
in full regulatory compliance. 


Key Indicator Rules (KIR) are those rules that are statistically 
predictive of overall regulatory compliance with all rules. These 
identified rules are generally in the mid-range of regulatory 
compliance and are very predictive between distinguishing 


those high-quality programs vs those that are not. 
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Quality Indicator Standards (QIS) are those standards that are 
Statistically predictive of overall program quality on various 
dimensions such as staffing, curriculum, parental involvement, 
and teacher behaviors in the classroom. 


Complaints can be any indications that there are issues at the 
specific facility that a concerned individual is reporting to the 
state licensing agency which require follow up and an 
abbreviated inspection review. 


Key Indicator Criteria are the specific criteria which make 
programs eligible for a Key Indicator Abbreviated Inspection. 
Examples of Key Indicator Criteria are the following: no 
change in director, less than 10% enrollment change, less than 
20% staff turnover, no change in corporate sponsorship, etc... 

And lastly, Compliance History should either demonstrate a 
very low level of non-compliance or a constant regulatory 
compliance improvement over time. See following equation. 


LQS = ©RCS + YRAR + /KIR + Y-QIS + ¥'Complaints 
+)'KI Criteria + );Compliance History 


The RCS should have a score either at a 7 or 5 level, Full or Substantial 
regulatory compliance. This should occur both at the aggregate and 
individual rule levels. 

The RAR should have no violations. 

The KIR should have no violations. 

The QIS should have a score in the range of 28-36+ on the Quality 
Scale. 

There should be no complaints about the program. 

All KI Criteria should have been met. 

And the Compliance History should have very few non-compliances 
and always be improving 
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Differential Monitoring x Integrated Monitoring 
Matrix 


Presented below is a proposed matrix depicting the relationship 
of integrated monitoring (IM) and differential monitoring 
(DM). Both integrated monitoring and differential monitoring 
have been discussed separately in previous posts. This 2 x 2 
matrix provides a visualization of how the two approaches 
potentially intersect and can be used in tandem. Just as a 
reminder, differential monitoring involves doing an abbreviated 
inspection instead of a full licensing inspection utilizing either 
a risk assessment or a key indicator predictor methodology. 
Integrated monitoring is the infusion of quality elements into a 
given set of rules or regulations, most likely through the use of 
Caring for Our Children. 


The 2 x 2 matrix provides four possibilities: A = Regulatory 
Compliance (RC) rules which results in a full inspection; B = 
Program Quality (PQ) standards which results in a full 
inspection; C = Regulatory Compliance rules which results in 
an abbreviated (Abb) inspection; and D = Program Quality 
standards which results in an abbreviated inspection. The 
essence of any model should be its relevance and hopefully its 
elegance. The below 2 x 2 matrix is relevant because the two 
monitoring approaches are the most salient ways of conducting 
inspections for human services regulatory administration. But 
hopefully it is also elegant in its simplicity and direct modeling, 
that we will need to see if it resonates with licensing 
administrators & researchers as well as regulatory scientists. 
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This matrix should help licensing administrators think through 
the appropriate use of these various approaches and what it 
means when combining them. Differential monitoring is an 
encouraged approach via CCDBG/CCDF, _ integrated 
monitoring is too new to make a determination regarding its 
use. I think it is the next evolution of program monitoring 
related to regulatory science and administration by providing a 
balance and continuum along the quality domain with 
regulatory compliance/licensing as the foundation of this 
continuum. TRLECE: The Role of Licensing in Early Care 
and Education has developed a wonderful research brief on 
program monitoring which highlights how states are using 
differential monitoring that I highly recommend (The Report). 


IM 
RC PQ 
DM Full A B 
Abb C D 
IM x DM Matrix 


Also, you may want to consult Licensing Measurement and 
Monitoring Systems: Regulatory Science Applied to Human 
Services Regulatory Administration which has a chapter about 
integrated monitoring (Licensing Measurement and Monitoring 
Systems ebook) 
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The below graphics, figures and displays help to support and 
depict various portions of text in Chapters 1-7. 


They depict the regulatory compliance theory of diminishing 
returns, the overall regulatory compliance and program quality 
model as described by Gwen Morgan, the ECPQIM: Early 
Childhood Program Quality Improvement and Indicator Model 
that depicts the relationship of regulatory compliance and 
program quality monitoring systems; a brief logic model of 
when risk assessment and key indicator methods can and cannot 
be used; the RAM: Risk Assessment Matrix decision matrix; 
the relationship of comprehensive reviews and abbreviated 
reviews, such as, risk assessment and key indicator reviews; 
data distributions for regulatory compliance, and program 
quality as depicted with QRIS and ERSs; ECPQIM theory; Key 
indicator and non-compliance relationship depicting the 
relationship between effectiveness and efficiency; the key 
indicator and risk assessment methodologies within a single 
matrix format; the use of Caring for Our Children in depicting 
the relationship between compliance and quality; the 
Regulatory Compliance Scale; the absolute and differential 
regulatory paradigms key elements; International study of child 
care comparing rules and regulations; Key indicator formula; 
examples of two data distributions from Head Start (skewed) 
and ECERS (normally distributed); ECPQIM version 5 which 
demonstrates the use of integrated monitoring. 


There are also a series of figures that summarize the contents of 
this ehandbook and have been used in various formats for 
webinars and presentations. And, finally, there is a trilogy of 
matrices that depict the key logic of KIM and RAM and are key 
to several of the RIKINotes posts in the Appendices as well as 
key indicator examples. 
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Figures, Graphs, Charts, Displays 


REGULATORY COMPLIANCE LAW OF DIMINISHING RETURNS 


5 
: 
8 


Substantial 


92 o4 


Regulatory Compliance Levets 


Methods for Achieving Quality Child Care 


GOALS 
SSS 
NONREGULATORY METHODS REGULATORY METHODS 
Public Education Accreditation/CFOC 
Training of Credentialing 
Dir i ag . Rate Setting —— | Fiscal Regulation 
a Quality Rating & 

Association Improvement 
Membership Systems 
ice ae Seeks Environmental || Licensing or || Building & 

cout Health Registration || Fire Safety 
Resource & Base line or floor of quality below which no service may legally operate 
= Exempt ‘i illegal Unlicensed 

Programs Criminal Operations 


Revised from YOUNG CHILDREN 
Vol. 34 No. 6 Sept. 1979, pp. 22-27 
Gwen G Morgan and updated by 
Rick Fiene, Dec 2012. 


Sanctions 


Abuse & Neglectful Care 
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Cl Visit— 
less than 
100% on KI 


— &RA 


7 5 Differential 
3 7 j_____* _ «Monitoring (DM) 


KI Visit— 
400% on 
previous KI 
&RA 


cx Yoo Vers Pars Yous Yr0 9 co 
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When Key Indicators and Risk Assessments Can Be Used 
The Licensing Law: 
All Rules that are promulgated based upon the Law 
Compliance Decision: Compliance Decision: 
100% compliance with all rules all the Substantial (96-99%) but not 1009 
time. compliance with all rules all the time. 

Key Indicators Risk Key Indicators Risk 

are ok to use Assessment are ok to use. Assessment 
cannot be ok to use. 
used. 

40 
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Fewer 
visits, key 
rules 


Risk Assessment Matrix (RAM) 


82 


*Regulatory Compliance 


Risk Assessment (RA) Matrix Revised {RC)(Prevalence/Probability /History + 


ewes High Medium ise Risk/Severity Level) 
Immediate 9 8 7 Tier 1 = ((RC= 93 —97) + (LowRisk)); ((98 — 99) + 
Short-term 6 5 a (Low Risk)) = Tier 1 
Long-term 3 2 1 

Probability 
Regulatory | 8+ rules outof | 3-7 rules out of 2 or fewer rules Tier 2=(RC=92 or less) + (Low Risk) = Tier 2 
Compliance | compliance. C liance, out of compliance. 
ter Hof | 92 or less oe regulatory | 98-99 regulatory Tier 3 = ((RC= 93 ~ 97) + (Medium Risk); (98 - 

Rules out of | regulatory compliance. compliance. 99) + (Medium Risk)) = Tier 3 
compliance | compliance. 

and In 
compliance Tier 4 =(RC=(92or less) + (Medium Risk)) =Tier 


4; (( 93 -97) +(High Risk) = Tier 4; ((98 - 99) + 
(High Risk)); ((92 or less) + (High Risk)) = Tier 4+ 


Using RAM to make licensing decisions 


Relationship of Comprehensive 
Reviews (CR) to Key Indicator (KI) or 
Risk Assessment (RA) Rule Non - 
Compliance 


Risk Assessment Rule 


Both 


Key Indicator Rule 


Risk to Children 


Prediction 
Non-Compliance Non-Compliance Non-Compliance 
2+ Rules = CR 1 Rule =CR Point System =CR 


1 Rule = Section 
Absolute scoring 1/0 


211 


1 Extreme Rule = CR 
Relative scoring 1/9 


ERS, QRIS, Licensing Comparisons 


ERS, QRIS, Licensing Distributions 


137 
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Key Indicator/Non-Compliance Relationship 


oO ee 
Key Indicator (blue)/Non-Compliance (gold) 


Frequency 


Effective Efficient 


KIM (Key Indicator Matrix) and RAM (Risk Assessment Matrix) 
Matrices Integration Into One Platform 


SS 


a 3 low 


Compliance 1 

Non-Compliance 4 5 5 Medium 
7 8 9 High 

Prevalence: low Medium High RAM 


This technical research note will integrate the Key Indicator Matrix (KIM) and the Risk Assessment 
Matrix (RAM) into one platform to clearly demonstrate their statistical modeling overldfey 

Indicators deal with the ability to predict overall compliance or performance based on existing 

data. RiskAssessment Indicators do not predict but determine a risk score based upon prevalence 
and severity measures Their purposes are different but when integrated together the two matrices 
are a powerful tool in determining the health of the measured entity. 


The above matrix integrates the two matrices of KIMand RAMand shows that KIMscores are 
generally at the lower end of risk but having sufficient prevalence when it comes to non 
compliance.RAM scores have a larger variance and are most concerning at the higher end of the 
continuum 
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Relationship of Health and Safety Rules/Regulations, Standards, and Guidelines in Early Care and Education 
by using the Caring for Our Children Publications 


ASPE 
Key Indicators .13 
Standards 
Caring for Our Children: Basicsas the risk 
assessmentkey indicator tool. 55 Standards. 


Stepping Stones _as the risk assessment tool based 
upon morbidity/mortality. 138 Standards . 


Caring for Our Children standards/guidelines as the comprehensive set of health and safety 
standards/guidelines for the early care and education field. 650 Standards . 
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Regulatory Compliance Scale (RCS) 
ST 


Regulatory Definitions & Number of 
Compliance Scale | Compliance Levels | Rule Violations 
Levels 
7 Full 100% 0 Violations 
Compliance 
5 Substantial 1-3 Violations 
Compliance 
3 Mediocre 4-9 Violations 
Compliance 
1 Low/Non- 10+ Violations 


Optimal Compliance 


214 


Regulatory Paradigms 


Absolute (Class, 1957) Relative/Differential (Fiene, 1985) 


Oo All rules are created co All rules are not 
equal. created equal. 
40 100% Compliance = o Full 100% + 
Full License. Substantial Compliance 
o PC + PQ = Linear. cuRpInricense: 


o All rules are 
vaviawed all he tine o Selected key rules are 


co PC + PQ = Not Linear. 


reviewed all the time. 


International Study of Child Care Rules (Fiene, 2013a) 


Inservice 
Pre 
Teacher 
Director 
GS 

ACR 


Parents ee 
Health (ERS 
Devel es 
lac i 


USA vs World 


————— ™ Countries 
—_—_—_____— 

———— 

—— 

0 0.5 1 15 2 
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Key Indicator Statistical Methodology 
a 


~ = (A)(D)-(B)(Q) = V (WXYZ) 


A= High Group + Programs in Compliance on Specific Compliance Measure. 
B = High Group + Programs out of Compliance on Specific Compliance Measure. 
C = Low Group + Programs in Compliance on Specific Compliance Measure. 
D=Low Group + Programs out of Compliance on Specific Compliance Measure. 


W = Total Number of Programs in Compliance on Specific Compliance Measure. 
X = Total Number of Programs out of Compliance on Specific Compliance Measure. 


Y = Total Number of Programs in High Group. 
Z = Total Number of Programs in Low Group. 


Head Start Performance Standards 
i 


100°) Mean = 3.33 
Std. Dev. = 3.769 
N=422 


Frequency 


Cl Total Violations 
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ECERS Total Scores 
OT 


Frequency 


3.00 4.00 500 600 7.00 
ecers total score 


ECPQIMS: Early Childhood Program Quality 
Improvement/Indicator Model Version 5 


Training/Technical Assistance/Coaching/Mentoring= 
Professional Development Quality Initiatives 
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Introduction to Licensing Measurement 


* The need for addressing licensing measurement and monitoring 
systems. Why now? 


* Regulatory science is a relatively new science. 

* Regulatory science, the FDA, and the medical arena. 

* History of licensing measurement. 

* History of standards/rule development in early care & education. 
* NARA's Licensing Curriculum. 

¢ NARA's Course on licensing measurement and systems. 


* The bottom line: Licensing data are very unique, not like most of the 
social science data we encounter. 


Conceptual/Theoretical Foundation 


* Regulatory compliance theory of diminishing returns. 

* Differential monitoring. 

* From theory to conceptual. 

* Methods for achieving quality child care model. 

* Early childhood program quality improvement & indicator model. 
* Regulatory compliance paradigms: Absolute vs Differential. 

* Ten elements of regulatory compliance paradigms. 

* The balancing act. 
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Ten Elements of Regulatory Compliance 
Paradigms 


* 1) Substantial versus Monolithic. 

* 2) Differential Monitoring versus One size fits all monitoring. 

¢ 3) Not all standards are created equal vs All standards are created equal. 
* 4) “Do things well” versus “Do no harm”. 

¢ 5) Strength based versus Deficit based. 

* 6) Formative versus Summative. 

¢ 7) Program Quality versus Program Compliance. 

* 8) 100-0 scoring versus 100 or 0 scoring. 

* 9) QRIS versus Licensing. 

* 10) Non-Linear versus Linear 


Regulatory Compliance & Program Quality 


* Quality initiatives 

* Quality rating and improvement systems. 
* Accreditation. 

* Professional development. 


* Relationship of regulatory compliance and program quality based 
upon the regulatory compliance theory of diminishing returns. 


* The ten elements of regulatory compliance and program quality 
continuum. 


* Implications for monitoring systems. 
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Principles of Instrument Design 


* Anecdotal & case record keeping. 

* Introduction of instrument-based program monitoring. 
* Reliability. 

* Validity and validation studies. 

* Statistical methods. 

* Data bases. 

* Nominal data measurement. 

* Nominal to ordinal measurement. 

* Lack of variance in the data. 

* Need for weighting. 

* Limitations of nominal measurement. 


Ten Elements of Regulatory Compliance and 
Program Quality Continuum 


* 1) “Do no harm” versus “Do good”. 

* 2) Closed system versus Open system. 

¢ 3) Rules versus Indicators. 

¢ 4) Nominal versus Ordinal measurement. 
¢ 5) Full versus Partial compliance. 

* 6) Ceiling effect versus No Ceiling effect. 
* 7) Gatekeeper versus Enabler. 

* 8) Risk versus Performance. 

* 9) Structural versus Process Quality. 

* 10) Hard versus Soft Data 
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Evolution of Monitoring Systems 


* Compliance monitoring, process monitoring. 
* Coordinated monitoring systems. 

* Qualitative monitoring systems. 

¢ Instrument-based program monitoring 

* Differential/Inferential program monitoring. 
* Key indicator approach. 

* Risk assessment approach. 


* Integrative program monitoring: Regulatory compliance x quality. 


What Research Tells Us and Doesn't 


* Idiosyncracies of licensing data. 

* Skewed distributions and potential reasons why. 

* Ceiling/plateau effect. 

* Curvi-linear/non-linear data vs linear data: Common assumption. 
* The dichotomization of data, why it is warranted. 

* Limitations of nominal data measurement. 

* Dealing with false negatives and false positives. 

* The need for validation studies. 

* Exploring regulatory compliance and quality interactions. 

* International data base is available for researchers. 
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Future Directions 


* Continue validating monitoring systems. 

* Nominal to ordinal measurement. 

* Balance between efficiency and effectiveness. 

¢ Balancing act between regulatory compliance and quality. 

* Continued development and validation of quality indicators. 


¢ Further development of the international data base of regulatory and 
quality indicators. 


* Continued development of statistical methods to deal with skewed data 
distributions, false negatives, and the other licensing data idiosyncracies. 


* Ability to better distinguish between the high quality performers 
and mediocre performers because of the ceiling/plateauing effect. 


Licensing and Quality Scale (LQS) (Fiene, 2023) 


- LQS = RCS + RAR + KIR + QIS + Complaints + KI Criteria 
+ Compliance History 


¢ The RCS should have a score either at a 7 or 5 level, Full 
or Substantial regulatory compliance. 


¢ The RAR should have no violations. 
¢ The KIR should have no violations. 


¢ The QIS should have a score in the range of 28-36+ on 
the Quality Scale. 


* There should be no complaints about the program. 
¢ All KI Criteria should have been met. 


- And the Compliance History should have very few non- 
compliances and always be improving. 
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KIM Matrix 


KIM Generator 


Rule In 
Compliance 


Rule Out of 
Compliance 


RAM Matrix 
High Risk/High 
Likely 


Med Risk/High 
Likely 


Low Risk/High 
Likely 


High Compliant 
Group 


High Risk/Med 
Likely 
Med Risk/Med 
Likely 
Low Risk/Med 
Likely 


KIM/RAM Validation Matrix 


KIM/RAM 
Validator 


KIM/RAM In 


Compliance 


KIM/RAM Out 
Compliance 


Rules In 
Compliance 
Yes/Yes: OK 
KIM/RAM 
No/Yes: False 
Positive 
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Low Compliant 
Group 


Yes: OK 


High Risk/Low 
Likely 


Med Risk/Low 
Likely 


Low Risk/Low 
Likely 


Rules Out of 
Compliance 


Yes/No: False 
Negative 


No/No: OK KIM 


Examples of Health and Safety Key Indicators 


Program is hazard free in-door and out-doors. 
Adequate supervision of children is present. 
Qualified staff. 

CPR/First Aid training for staff. 

Hazardous materials are inaccessible to children. 
Staff orientation and training. 

Criminal Record Checks. 

Ongoing monitoring of program 

Child immunizations 


Examples of Key Indicator Applications 


e Health and Safety Licensing Key Indicators planned or 
implemented in the following states and provinces: Pennsylvania, 
Kansas, California, Illinois, Indiana, West Virginia, Michigan, 
Ontario, British Columbia, Saskatchewan, Montana, Oregon, 
Washington, New York, Maine, Texas. 


e Stepping Stones Key Indicators 

e Office of Head Start Key Indicators. 

e Accreditation Key Indicators - NECPA — National Early 
Childhood Program Accreditation. 

e Environmental Rating Scale Key Indicators — Centers. 

e Environmental Rating Scale Key Indicators — Homes. 

e Caregiver Interaction Scale Key Indicators. 

e Quality Rating & Improvement System Key Indicators — 
QualiStar. 

e Footnote: Child & Adult Residential Care Key Indicators. 
e Footnote: Cruising Industry in general and Royal Caribbean 


in particular. 
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This last section has an example validation report, 


the UCM research abstract, timeline, and glossary 
of terms 


The Saskatchewan Early Care and Education Quality 
Indicators Tool and Validation: The Last Piece of the 
Puzzle in Creating a Differential Monitoring Approach 


Abstract 


This validation study involved 30 programs, 90 classrooms and 180 
observations of infant, toddler, and preschool classrooms utilizing the 
ECERS/ITERS and the SKECPQI instruments. Six trained observers 
collected the data over a two-month period. The analyses clearly 
demonstrated that the new SKECPQI instrument is a valid and 
reliable measure of program quality. PQI #2 clearly showed it 
predictive power in this study. The SKECPQI and PQI #2 correlated 
very highly with the ITERS and ECERS. The SKECPQI appears to 
correlate more highly with regulatory compliance violations than the 
ECERS or ITERS. The ceiling/plateauing effect is not as evident 
with the SKECPQI as it is with ECERS/ITERS. The Regulatory 
Compliance Scale (RCS) is a better sorter for regulatory compliance 
than the violation data. There is a good deal of internal consistency 
within the SKECPQI Tool just as it is with the ERSs. The Regulatory 
Compliance Theory of Diminishing Returns was validated in 
comparing RCS with ECERS/ITERS. Both the SKECPQI Scale and 
the Regulatory Compliance Scale are introduced as new 
improvements to measuring quality and regulatory compliance. 
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INTRODUCTION 


This article will delineate the development, piloting and 
validating of the Saskatchewan Early Care and Education 
Quality Key Indicators (SKECPQI) Tool. The purpose of the 
tool is to assess the overall program quality in centered based 
childcare programs in the Province of Saskatchewan, Canada. 
The evolution of the tool resulted from a multi-year effort by 
the Ministry of Education in the Province of Saskatchewan to 
build an effective and efficient differential monitoring system. 
This effort in building a new differential monitoring system 
started in 2019 and was completed in 2023. The first 
component of this restructuring was the Saskatchewan 
Licensing Key Indicator System (2019). This was followed by 
the Saskatchewan Risk Assessment Rules (2019). Once these 
were in place and operational, a validation study was conducted 
to measure that the two methodologies were operating as they 
should (2020). A work group was initiated in 2019 and 
completed its work in 2020 on an Early Care and Education 
Quality Key Indicator Tool (SKECPQI). The tool was put on 
hold for 2021 because of the pandemic and a new Canadian 
Federal initiative to expand childcare services across the 
province. The tool initiative began again in 2022. The pilot 
testing and validation occurred in 2023. 


The work and these studies in the Province of Saskatchewan by 
the Ministry of Education is the first demonstration of a full- 
blown differential monitoring system involving licensing key 
indicator rules, risk assessment rules, and quality indicators. 
Besides the development of each tool, each of these tools have 
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been validated as well. All this work was done as a 
collaborative effort between the Ministry of Education staff and 
the National Association for Regulatory Administration 
(NARA) consultant pool. Presently, Saskatchewan’s overall 
system is the best example of a fully developed differential 
monitoring system for the early care and education field. 


This was a monumental effort involving many individuals at the 
local, provincial, and national levels and many hours of data 
collection and analysis. All the reports are available on the 
NARA Website (https://www.naralicensing.org/key- 
indicators) and the full data set will be available via Mendeley 
Data Sources (https://data.mendeley.com/datasets/kzk 
6xssx4d/1). 


BACKGROUND HISTORY 


This study and tool grew out of an interest by Saskatchewan 
Ministry of Education policy makers to establish a balance 
between regulatory compliance and program quality in the most 
effective and efficient manner. The Province of Saskatchewan 
did not have a QRIS (Quality Rating and Improvement System) 
in place nor plans on developing one. Generally, when a 
jurisdiction wants to develop a balance between regulatory 
compliance and program quality with 
rules/regulations/standards, QRIS’s are generally developed 
and implemented. 


In reviewing the research literature on regulatory science, 

differential monitoring has been a developing approach used by 

many other jurisdictions in the human service licensing field, 

especially in the United States and in several other Canadian 
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Provinces. Based upon this review of the research literature and 
the work of the National Association for Regulatory 
Administration (NARA) which has been a long-term promoter 
of this approach and the resulting methodologies of licensing 
key indicators, risk assessment rules, and most recently quality 
indicators, a contract was entered into between the Ministry of 
Education and NARA. 


The tool is the direct result of research into identifying licensing 
and quality key indicators over a 50-year (1970-2022) research 
effort in which specific methodologies were developed and the 
differential monitoring approach was tested and implemented 
in the 1970’s. Since that time, a national database which 
expanded to an international database of common key 
indicators from jurisdictions’ respective key indicator tools. 
These key indicators resulted in a very similar tool that 
Saskatchewan is using. In fact, in 2019 when the Saskatchewan 
work group was established, they started with that specific tool 
that had been developed (Fiene, 2019). During the 2019-2020 
period, the work group made the tool into a more user-friendly 
tool for Saskatchewan childcare programs. 


The big deal with utilizing the key indicator methodology is its 
ability to statistically predict as if one administered the full tool 
in question. Therefore, when one administers the first quality 
indicator in the Saskatchewan Early Care and Education 
Quality Indicator tool, it is as if they have administered a 
licensing based regulatory compliance instrument since the 
quality of staff is a statistically predictive rule (Fiene, 2002a). 
The same is true in administering the curriculum quality 
indicator because it is a statistically predictive standard when 
looking at overall program quality (Fiene, 2002b). When it 
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comes to QRIS, having communication between staff and 
parents and parental involvement is a statistically predictive 
standard for an overall set of QRIS standards (Fiene, 2014). 
And finally, when administering the ECERS and ITERS or the 
CIS quality item indicators these are all statistically predictive 
items for their respective scales as if you had administered the 
full scales (Fiene, 2002b). 


So, as a state/provincial administrator, I would be interested in 
focusing my efforts on these indicators which reflect 
compliance with high quality rules/regulations/standards for 
early care and education. This would be my starting point. I 
would make sure that my standards reflected quality teachers 
with the necessary supports such as coaching/mentoring, an 
early care and education philosophy based upon an emergent 
curriculum where children are viewed as competent learners, 
developmentally appropriate curriculum and child assessments, 
parental and staff communication and participation, and teacher 
language based/communicative focus when interacting with 
children in a give and take manner. All this done within a warm 
and loving style. 


An even more efficient and effective way of using the new 
program quality tool is to pair it with the National Center for 
Health and Safety in Child Care’s Parental Guide to Choosing 
Safe and Healthy Child Care (DHHS: Assistant Secretary’s 
Office for Planning and Evaluation, 2019). This is a more 
aggressive and controversial approach, but it is the most 
efficient way of conducting monitoring visits in the most 
abbreviated way. However, as_ efficiency increases, 
effectiveness may decrease; so, it is a delicate balancing act. 
This suggested approach builds off a similar suggestion in 
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which only using Caring for Our Children: Basics (ACF, 2015) 
a DHHS Administration for Children and Families publication 
would be used as the base for regulatory compliance in the 
United States. 


Differential monitoring grew out of a need for jurisdictions to 
be more effective and efficient in their oversight and inspection 
efforts of early care and education programs. This started to 
occur in the late 1960’s and 1970’s as many more programs 
were being established. It was becoming clear that the old one 
size fits all approach to program monitoring was being 
overwhelmed by the increasing numbers of programs. Also, 
from an efficiency standpoint it did not make sense to spend the 
same amount of time with programs that were performing well 
as those that really needed additional attention. The birth of 
differential monitoring occurred which at that time it was called 
inferential inspections (Fiene & Kroh, 2000). Different 
terminology, same concept. 

Since then, differential monitoring has two basic methodologies 
that have been used successfully over the years: risk assessment 
and key indicators. The two methodologies have the same 
results, shortened or abbreviated reviews but they differ in their 
approaches. Risk assessment as the name implies identifies 
specific standards that place clients/children at greatest risk or 
morbidity or mortality if not complied with. Key indicators are 
specific standards that statistically predict overall regulatory 
compliance with all rules. Each has their place in the 
differential monitoring approach depending on _ the 
jurisdictions’ emphasis. Most recently, to balance the emphasis 
on regulatory compliance has been the introduction of quality 
indicators which are specific standards drawn from quality 
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initiatives, such as professional development, program quality 
tools, and quality rating & improvement systems. 


It is and always has been recommended that these 
methodologies be used together and not separately. This final 
study undertaken in the Province of Saskatchewan completes 
the cycle of doing just that in developing a fully functional 
differential monitoring system with key licensing and quality 
indicators as well as risk assessment rules. 


THE STUDY DESIGN AND METHOD 


The design of this study was to provide a validation study of the 
use of the Saskatchewan Early Care and Education Quality Key 
Indicators Tool. A convenience sample was selected in which 
a good variation of overall quality would be present. There 
were to be three buckets of quality: High, Middle, and Low. 
These would be defined via ERS scores. Because this was a 
validation study it was critical to have sufficient variation in the 
overall quality of programs to test the sensitivity of the new 
assessment tool. 


The below table (Table 1) provided the guidance to the 
Saskatchewan Ministry of Education policy staff in determining 
how to collect the program quality data for the research pilot 
study related to early childhood quality indicators. 
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Table 1: Selection Process for Study Programs 
Quality | Center Levels 


High 10 Infant 
Toddler 
Preschool 

Middle Infant 
Toddler 
Preschool 


Infant 
Toddler 
Preschool 


MN 

EF 

WIN] Re} ]rNml] Re] Ww] po] e < 
— 

Q 


QQ), BS al we) Sl) al) aS 


Notes: 


A =ITERS (Infants) (B-lyr) 

B = ITERS (Toddlers) (lyr-2yrs) 

C = ECERS (Preschoolers) (3+yrs) 

1 = SKECPQI/infant (QI items 1-5, 7, 9-10) 

2 = SKECPQI/Toddler or Preschool (QI items 1-5, 7, 9-10) or (QI items 1-6, 8-10) 
3 = SKECPQI/Preschool (QI items 1-6, 8-10) 

SKECPQI = Saskatchewan Early Childhood Program Quality Indicators tool 


A total of 6 trained data collectors were needed, 3 for the ERSs 
and 3 for the SKECPQI. Each observer collected data from 30 
classrooms. A data coordinator was utilized who collected all 
the data, reviewed the scores from the various tools and sent 
them to NARA. The data collectors were not aware of which 
centers are in which group, such as High, Middle, or Low. 


See the Appendix for the Draft of the SKECPQI tool that was 
used during data collection. 
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As said earlier, this study involves the validation of the 
Saskatchewan Early Childhood Quality Indicators Tool 
(SKECPQI) and involved the collection of new data utilizing 
the new tool and collecting Early Childhood Environmental 
Rating Scale (ECERS/ITERS) data as well. Independent 
contract staff were trained in the use of the SKECPQI as well 
as having had training on the ECERS/ITERS and were 
proficiently reliable on the ECERS/ITERS. 


A sample of 30 childcare programs who volunteer to be part of 
this study was selected with 1/3 identified as high quality, 1/3 
identified as medium quality, 1/3 identified as low quality. 
Each program had both the SKECPQI and the ECERS/ITERS 
administered to them utilizing two independent observers. The 
data from the SKECPQI was compared to the ECERS/ITERS 
to determine the relationship between the two/three scales. The 
research hypothesis is that there will be a positive relationship 
between the two/three scales in which those programs that score 
high on the SKECPQI will score high on the ECERS/ITERS 
and those that score low on the SKECPQI will score low on the 
ECERS/ITERS. The ECERS/ITERS will be used as the 
reference tool for establishing the validity of the SKECPQI. 


A training program and all necessary revisions to policies and 
procedures was conducted as part of this project by a NARA 
Consultant on both phase 1 and 2. It will be determined later if 
the SKECPQI will be administered on an ongoing basis by 
contracted staff or by Ministry staff. Reporting templates were 
developed as part of this implementation stage. The 
implementation stage was evaluated to make certain that all 
components are in place and working as they should. 
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Timeline: Phase 1: 6 months; Phase 2: 9 months; Training and 
Implementation Phase: 12 months, will overlap with phase 1 
and 2 and extend beyond both. The total time frame will be 24 
months (about 2 years), this will include the final report and 
final evaluation of the implementation stage 


RESULTS 


The ECERS and ITERS were used to validate the new 
Saskatchewan Early Care and Education Quality Indicators 
Tool (SKECPQI). This is standard procedure when conducting 
a validation study, a recognized empirically based and accepted 
standard tool is used in correlational analyses to determine if 
the new tool is measuring the same dimensions as_ the 
standardized tool. 


The target tool, the Saskatchewan Early Care and Education 
Quality Indicators, was to be validated against the ECERS and 
ITERS to determine if there was a quality relationship between 
the two tools. 


The validation analyses involved detailed correlational analyses 
between the various scales to determine if a relationship existed 
and how strong that relationship was. But before delving into 
this relationship and these analyses, an additional analysis was 
performed given the sophisticated nature of the Saskatchewan 
monitoring system. Saskatchewan’s Ministry of Education’s 
designed differential monitoring system is by far the most 
analyzed of all jurisdictions to date, so it was suggested to take 
advantage of this level of detail and build in an additional series 
of analyses to further test the regulatory compliance theory of 
diminishing returns in conducting this study. By doing so, 
Saskatchewan joins the ranks of the Provinces of Alberta and 
Ontario, the US States of Georgia and Washington, and the US 
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National Head Start program in conducting studies to either 
confirm or not this theory of regulatory compliance (please see 
the NARA website on key indicators which contains all the 
research reports). The following results delineate the data from 
that portion of the study. 


As part of the data collection in addition to collecting data on 
the ECERS and ITERS as well as the Saskatchewan Early 
Childhood Program Quality Indicators scale, a summary sheet 
containing regulatory compliance data was also obtained on 
each program. These data contained essential demographic 
information as well as violations from the last inspection along 
with a rating of the program which was cross referenced to the 
regulatory compliance data to generate a Regulatory 
Compliance Scale. This Regulatory Compliance Scale (RCS) 
had four levels of regulatory compliance: Full, Substantial, 
Medium, and Low. This RCS is like the regulatory compliance 
structure used in the previous studies in the above-mentioned 
jurisdictions in the US and Canada and has been further 
developed as a more valid means for measuring and analyzing 
regulatory compliance (Fiene, 2022). In the Fiene RCS, the 
following rubric was used: Full = 0 violations; Substantial = 
1-3 violations; Medium = 4-9 violations; and Low = 10+ 
violations. 


The first set of analyses was to determine if a correlation existed 
between the RCS and the ECERS and ITERS. This was the 
case with the following results: RCS x ITERS for the infant 
classrooms = .54; p < .002; RCS x ITERS for the toddler 
classrooms = .42; p < .03; and RCS x ECERS for the preschool 
classrooms = .75; p < .0001. 
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The second level of analyses (ANOVA) was to determine if the 
RCS levels of Full, Substantial, Medium, and Low 
demonstrated any significant differences in the ECERS and 
ITERS. The results were the following: Infant classrooms: 
Low = 3.07; Medium = 4.89; Substantial = 5.06; Full = 4.69; F 
= 11.43; p < .0001. Toddler classrooms: Low = 3.50; Medium 
= 4.56; Substantial = 4.62; Full = 5.06; F = 2.27; p < .11. 
Preschool classrooms: Low = 2.78; Medium = 4.39; Substantial 
= 4.90; Full = 5.12; F= 16.27; p < .0001. Apart from the toddler 
classrooms, both the infant and preschool classrooms support 
the regulatory compliance theory of diminishing returns ceiling 
and plateauing effect when it comes to measuring program 
quality as one moves up the regulatory compliance scale. 


Table 2: Regulatory Compliance Scale (RCS) and 
ECERS/ITERS Scores 


RCS 


Infant 
Classrooms 


Toddler 
Classrooms 


Preschool 
Classrooms 


Low 


3.07 


3.50 


2.78 


Medium 


4.89 


4.56 


4.39 


Substantial 


5.06 


4.62 


4.90 


Full 


4.69 


5.06 


5.12 


Significance 


F=11.43 
0001 


F=2.27; 11 


F=16.27 
0001 


ECERS, ITERS for Infant classrooms, ITERS for Toddler 
classrooms (n = 90): 


The ECERS score ranged from 1.41 to 6.00. The ITERS for 

infant classrooms ranged from 2.16 to 5.77; and the ITERS for 

toddler classrooms ranged from 2.14 to 5.90. The respective 

means for the ECERS, ITERS-Infant classrooms, and the 
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ITERS-Toddler classrooms were the following: 4.09, 4.39, 
4.39. The means and ranges were all consistent. 


The correlations of the infant, toddler and preschool classrooms 
in each of the 30 facilities were the following: Infant and 
Toddler classrooms = .65; p < .0001; Infant and Preschool 
classrooms = .74; p < .0001; and Toddler and Preschool 
classrooms = .52; p < .005. The classrooms demonstrated a 
great deal of consistency across the various facilities which one 
would expect. 


SKECPOI for Preschool, Infant, and Toddler Classrooms (n = 
90): 


The SKECPQI score ranged from 13 to 100. The SKECPQI for 
infant classrooms ranged from 31 to 91 (Mean=60.10); the 
SKECPQI for toddler classrooms ranged from 13 to 100 
(Mean=55.07); and the SKECPQI for preschool classrooms 
ranged from 25 to 100 (Mean=57.48). 


The correlations of the infant, toddler, and preschool 
classrooms in each of the 30 facilities were the following: Infant 
and Toddler classrooms = .74; p < .0001; Infant and Preschool 
classrooms = .85; p < .0001; and Toddler and Preschool 
classrooms = .75; p < .0001. The classrooms demonstrated a 
great deal of consistency across the various facilities which one 
would hope to be the case with this type of tool or scale. Based 
upon these results, the inter-correlations were extremely high 
and show a great deal of stability and are a reliable measure of 
quality indicators. 


SKECPQI #2 showed a great deal of promise as a standalone 
quality indicator. SKECPQI#2 correlated significantly with 
ITERS (.56; p < .0001), and ECERS (.61; p < .0001) and with 
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the overall SKECPQI scores for infant classrooms (.88; p < 
0001), toddler classrooms (.81; p < .0001), and preschool 
classrooms (.90; p < .0001). This quality indicator dealt with 
philosophy, curriculum planning and programming. This is not 
the first time that such an indicator was an excellent predictor. 
This result has been the case in other program quality studies as 
well (Fiene, Greenberg, Bergsten, Fegley, Carl, Gibbons, 
2002b). 


The SKECPQI scale demonstrated a great deal of robustness in 
the data distribution and a good deal of variation in the data set. 
These are the characteristics of a new tool that you would hope 
to find in the scale construction and implementation. 


Regulatory Compliance Data for Each of the Programs (n = 
30): 


The Regulatory Compliance Scale (RCS) distributions were the 
following: Full = 13%; Substantial = 20%; Medium = 37%; and 
Low = 27%. Generally regulatory compliance data are more 
skewed than this distribution but because of the nature of this 
study, facilities were deliberately selected breaking them up 
into these categories/levels. 


The Regulatory Compliance Scale (RCS) actual regulatory 
compliance violations played out in the following table, these 
results for the average number of violations were statistically 
significant (F = 3.69; p < .03): 
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Table 3: Regulatory Compliance Scale by the Number of 


Violations 


Regulatory 
Compliance 
Means 


Number of 
Facilities 


Low 


4.75 


Medium 


3.90 


Substantial 


1.60 


Full 


0 


Comparing the ECERS and _ITERS with SKECPQI and 
Regulatory Compliance (RCS) Data: 


These are the correlations between RCS and SKECPQI for 
infants, toddlers, and preschool classrooms. RCS x PQI for the 
infant classrooms = .58; p < .001; RCS x SKECPQI for the 
toddler classrooms = .51; p < .005; and RCS x SKECPQI for 
the preschool classrooms = .60; p < .001. The SKECPQI 
clearly demonstrates its relationship with regulatory 
compliance. Also, when the SKECPQI is compared with 
regulatory compliance violation data, the correlations are 
higher than those obtained in comparing the ERSs to regulatory 
compliance violation data. And, in fact, the SKECPQI when 
compared with the RCS appears not to have a ceiling or 
plateauing effect. It would appear that the SKECPQI is 
measuring quality in a different way since this effect does not 
appear evident in the RCS distributions. This result will need 
to be confirmed in other studies to make certain this relationship 
holds up. This is a first for comparing regulatory compliance 
data with program quality data. In the past, either a ceiling or 
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plateauing effect was always present when looking at the 
relationship between regulatory compliance and program 
quality. 


Here are the correlations between SKECPQIs and ERSs for 
infant, toddler, and preschool classrooms: PQI x ITERS for the 
infant classrooms = .66; p < .0001; PQI x ITERS for the toddler 
classrooms = .53; p < .003; and PQI x ECERS for the preschool 
classrooms = .66; p < .0001. These inter-correlations most 
suggest that the SKECPQI is a valid tool measuring program 
quality on a different dimension (quality indicators) than the 
ERS but measuring quality, nonetheless. 


A regression analysis determined that with RCS as the 
dependent variable, ECERS and regulatory violations were 
statistically significant at the p < .0001 with an R = .91. This 
accounted for practically 75% of the variance in being able to 
determine regulatory compliance. 


DISCUSSION 


Last piece of the puzzle in creating a differential monitoring 
system, that is how this report is being characterized. The 
Province of Saskatchewan has undertaken all the other 
methodologies utilized in a differential monitoring approach 
(Please see the NARA website for these reports, the link is hot 
linked on the first page of this report). Licensing key indicators 
and risk assessment rules have been implemented successfully. 
What remained were the Quality Indicators. This report 
completes the full cycle of validating these last indicators. 


With the completion of this validation study, the Saskatchewan 
Early Childhood Program Quality Indicators Scale could be 
adapted by other jurisdictions and utilized as a screener 
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methodology. The reason for suggesting this approach is that 
all the quality indicators are taken from the Key Indicator 
Methodology and therefore have predictive value when it 
comes to determining overall quality (Fiene, 2019a). Also, the 
indicators are drawn from several early care and education 
delivery systems and quality initiatives, such as licensing, 
QRIS, quality scales, accreditation, and _ professional 
development. 


The other significant finding from this study was the additional 
confirmation of the regulatory compliance theory of 
diminishing returns in which the results from this study are 
consistent with the findings from other studies conducted in 
Canada and the United States. This continues to be a major 
finding when it comes to comparing regulatory compliance 
with program quality and the resulting ceiling and/or plateauing 
effect related to quality scores. Again, from a public policy 
viewpoint, this finding has significant implications in how 
licensing decisions are or should be made. 


A very interesting finding which was not expected was the fact 
that when the SKECPQI scores were compared with the 
regulatory compliance violation data the — usual 
ceiling/plateauing effect did not emerge as in previous studies 
when these types of analyses were performed. This result needs 
further exploration to determine why this occurred. In future 
studies utilizing the SKECPQI, it will be necessary to do similar 
analyses with regulatory compliance data to ascertain if this 
same result occurs. At this point, it is difficult to determine if 
it is characteristic within the SKECPQI that is producing this 
result, such as a better balance between regulatory compliance 
and program quality. Only with further study will we be better 
able to determine the cause of this different result. 
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CONCLUSION 


This article will be read with a certain amount of skepticism in 
that it suggests using differential monitoring on a much broader 
scale; however, this report is like several other validation 
studies conducted by NARA over the past decade which have 
now clearly demonstrated the validity of the differential 
monitoring approach. And because of these validation studies, 
the differential monitoring approach has been utilized by many 
jurisdictions and has been cited in the United States Federal 
Legislation that reauthorized the Child Care and Development 
Block Grant. In the legislation, it is suggested but not required 
that states entertain the use of the approach. Based upon the 
latest childcare licensing data, it appears that many states have 
attempted to utilize the approach. 


This study fits with the other regulatory compliance theory 
reports from states and provinces that have been completed over 
the past decade by NARA. As mentioned in the Results and 
Discussion Sections, this study is the most comprehensive of 
the group since the Province of Saskatchewan developed not 
only risk rules and key indicator rules for licensing but also 
quality indicators that could be used within their differential 
monitoring system. This is the first demonstration of this 
comprehensive approach. 


This study completes what was to be a three-year effort but 
turned into a five-year effort because of the COVID19 
Pandemic. Each component of this overall project is well 
documented on the NARA Key Indicator website. The three 
major results of this study: confirmation of the regulatory 
compliance theory of diminishing returns, the introduction of 
the regulatory compliance scale and the introduction of the 


242 


Saskatchewan Early Childhood Program Quality Indicators 
Tool/Scale are all significant contributions to the licensing 
research literature, but it is this last contribution that needs 
further development. 


The Saskatchewan Early Childhood Program Quality Indicators 
Tool/Scale is a new program quality tool that is rather robust in 
measuring quality using key indicators which are taken from 
various quality initiative studies conducted over the past several 
decades. The hope is that it will continue within the early care 
and education field being validated by other researchers and 
being used to determine the relative scope of program quality 
in various early care and education settings. We could see the 
scale being utilized throughout the United States and Canada. 
It would be an excellent supplement to either the ERS or 
CLASS tools. It is a simple, straightforward tool that can be 
easily trained on and administered. It could provide an 
interesting supplement for licensing staff when they are doing 
their licensing reviews. In fact, it is intended to be used in 
conjunction with licensing key indicators and risk rule tools. 


Although this was not reported in the Results Section, we think 
it is vitally important to highlight the significant contributions 
of the licensing staff and others who helped to develop the 
groupings and levels of regulatory compliance and quality. It 
was only because of their level of early childhood expertise and 
their knowledge of the programs that made the sequencing so 
effective and impactful as an analytical frame of reference. 


One last thought is the introduction of the Regulatory 
Compliance Scale (RCS) as a more logical and robust rubric 
when comparing regulatory compliance data with program 
quality. This thought has been presented elsewhere as a 
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possible improvement within licensing measurement and 
monitoring systems (Fiene, 2022). The scale has been piloted 
in the past, but this is the first formal test of it in a specific 
jurisdiction. 


For additional information regarding this research validation 
study, please contact: Richard Fiene PhD, Research 
Psychologist & Regulatory Scientist, Research Institute for 
Key Indicators, Penn State University, 


rfiene @rikinstitute.com 
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Saskatchewan’s Early Learning and Child Care Program 
Quality Key Indicator Instrument (SKECPQI) The 
Saskatchewan Program Quality Work Group? 


INTRODUCTION and BACKGROUND to SKECPQI 


Ten Quality Key Indicators (QKI) make up the Saskatchewan’s Early 
Learning and Child Care Program Quality Key Indicator Instrument 
(SKECPQI). The details about each of the Quality Indicators and 
data collection instructions in order to obtain the necessary data to 
determine if a program meets the Key Quality Indicators are 
delineated below for each quality key indicator. Part 1 - Quality 
Key Indicators (QKI) 1 —5 will be collected via record or document 
review, interviewing individuals, or observation. Part 2 - Quality 
Key Indicators (QKI) 6 — 10 will be collected via observations in the 
classrooms throughout the assessment. 


These ten quality key indicators were taken from previous studies 
conducted over the past 40 years by Dr Richard Fiene utilizing the 
Regulatory Compliance Key Indicator metric (RCKIm) that he 
developed in the late 1970’s. These QKI have held up over time 
and have now been coupled together into this tool and being pilot 
tested in the Province of Saskatchewan. The original tool was 
reviewed by a Provincial Ministry of Education Work Group who 
met during 2019-2020 and made some revisions to the original 
tool. All these changes are reflected in this version of the SKECPQI 
(2023). 


1) Saskatchewan Program Quality Work Group: Kim Taylor, Derek Pardy, Cindy 
Jeanes, Tanya Mengel, Samantha Ecarnot, Karen Heinrichs, Michelle 
Vellenoweth, Kristin Jarvis, and Rick Fiene. 
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PART 1 — Record/Document Review, Interview, Observation 
Quality Indicators 


INDICATOR 1): Number of ECE III Educators 


Assessors will review staff records to determine the number of 
staff who have these credentials in early childhood education. 
Record the number of ECEs with the appropriate qualifications and 
divide them by the total number of ECEs to come up with a percent 
for the center. 


How to Measure: 


Go to the Staff Information Summary form to obtain the data for 
this item. There are two columns that will do this. Under 
Certification: Certification Date and Certification Level (Highest ECE 
Level Certified). The certification date should be earlier than the 
date of the review and the actual level of the certification. In this 
case, we are interested in the number of (ECEIll's). Record the 
number of ECEIII working at least 65 hours/month. Then record 
the number of total teaching staff working at least 65 hours/month 
below as well. Teaching staff is defined as staff who have a 
responsibility for working with the children and the programming. 
Determine the percentage by dividing the total number of staff into 
the total number of ECEIII Certified teaching staff, ECEIII Certified 
teaching staff is the numerator, and the total number of teaching 
staff is the denominator (ECEIII|/Total number of teaching staff x 
100% = Percent). 


Scoring for PQI 1: 
The total number of ECEIII Certified teaching staff (1.1) 


The total number of teaching staff (1.2) 
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Total ECEIII teaching staff divided by the total number of teaching 
staff (%). 


Then based on the percentage, you can find the score of 1-4 as per 
the chart below. 


Circle the 1=Oto | 2=26to | 3=51 4=76to 
Appropriate Level 25% 50% to 75% | 100% 


INDICATOR 2): Stimulating and Dynamic Environment 


The criteria for measuring this are drawn from Play and 
Exploration Guide. The program is child centered. Children are 
viewed as competent learners, and they have the freedom to 
access classroom materials independently without adult 
intervention. The children are provided with meaningful choices 
through activity/learning centers. There is evidence of the 
children’s interests and their projects in the learning environment. 


How to Measure: 


Below is the checklist of items that should be present to assess if 
the environment is both stimulating and dynamic for the children. 
You will want to observe that the following items are occurring in 
the classroom first. If you do not actually observe it occurring, then 
check the program plan to find documentation that it normally 
occurs but you just did not observe today. The checklist items 
would be found in Play and Exploration foundational materials. 


Quality Early Learning Environments (Please record all that you 


observe Y or N): 


1. Co-teaching is evident. Y/N (2.1) 

2. Children are viewed as competent learners & can access 
materials independently. Y/N __ (2.2) 

3. Authentic and meaningful materials are used with 
children. Y/N (2.3) 
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4. Children are provided with meaningful choices. Y/N 


(2.4) 

5. Children’s work, art and photos are displayed respectfully. 
Y/N (2.5) 

6. Family photos are displayed in the early learning program. 
Y/N (2.6) 

7. Documentation of learning is displayed and discusses 
holistic development. Y/N (2.7) 

8. Environment reflects the culture and beliefs of the 
children, families and staff. Y/N (2.8) 

9. Variety of books & other print materials are available 
throughout the classroom Y/N ___ (2.9) 

10. A variety of writing materials are accessible to children 
most of the time. Y/N (2.10) 


11. There is evidence of the children’s interests & projects in 
the classroom. Y/N___ (2.11) 


Scoring for PQI 2: 


Total up the number of items where you recorded a “Y” above that 
you observed (curriculum or in classrooms), divide by 11 x 100% to 
come up with a percent and record here %. 
Then based on the percentage, you can find the score of 1-4 as per 
the chart below. 


Circle the 


Appropriate Level 


INDICATOR 3): Developmentally Appropriate Curriculum 
Based on Assessments of Each Child 


The key for this quality key indicator is that the program is 

following an individualized prescribed planning document when it 

comes to curriculum. It does not mean it is a canned program, in 

fact, it shouldn’t if it is based upon the individual needs of each 

child’s developmental assessment. The assessor will ask to see 
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what is used to guide the curriculum. There should be a written 
document that clearly delineates the parameters of the philosophy, 
activities, guidance, and resources needed for the particular 
curricular approach. There should also be a developmental 
assessment which is clearly tied to the curriculum. The 
developmental assessment can be home-grown or a more 
standardized off-the-shelf type of assessment, the key being its 
ability to inform the various aspects of the curriculum. The 
purpose of the assessments is not to compare children but rather 
to compare the developmental progress of individual children as 
they experience the activities of the curriculum. 


The following key elements should be present when assessing this 
quality indicator. 


e 1) The program practices emergent curriculum, allowing 
the interests of the children to determine the learning 
content. The curriculum is informed by individual 
developmental assessments of each child in the respective 
classrooms. 

e 2) The children and educators are co-learners in the 
exploration of projects. 

e 3) Learning activities of the children are documented, 
displayed in the learning environment and used to plan 
further learning activities. This can be assessed 
developmentally. 


How to Measure: 


Take a sample of 10 individual children's records and consider the 
above three elements for EACH record. You should be asking 
yourself if there is a clear link between an assessment and the 
developmentally appropriate curriculum so that an individualized 
learning approach is being undertaken and each child's 
developmental needs are taken into consideration. These records 
could be formal, such as portfolios kept for each child or a more 
informal, anecdotal type of record keeping. The key is that there is 
a record that can be looked at. It is not adequate if the teacher 
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says they do it from memory — it needs to be written down and 
documented. 


Cross check the child's record to the actual curriculum. Record all 
the instances (Y’s) in which this occurs. All three blocks need to be 
checked for each record (1-10). 


Emergent Curriculum is Practiced (3.1) 


1 2 3 4 5 6 7 8 9 10 
Y/N | Y/N | Y/N | Y/N | Y/N | Y/N | Y/N | Y/N | Y/N_ | Y/N 


Key Element 1 + 


Children and Educators are Co-learners (3.2) 


1 2 3 4 5 6 7 8 9 10 
Y/N | Y/N | Y/N | Y/N | Y/N | Y/N | Y/N | Y/N | Y/N_ | Y/N 


Key Element 2 + 


Learning Activities are Documented and Displayed and Used to 


Plan Future Learning (3.3) 


1 2 3 4 5 6 7 8 9 10 
Y/N | Y/N | Y/N | Y/N | Y/N | Y/N | Y/N | Y/N} Y/N_ | Y/N 
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Key Element 3 + 


All three key elements must have a Y to get an overall score of Y. If 
all three key elements have a Y for that individual record, then 
record Y in the corresponding block in the overall score. 


1Ys |2Ys |3Ys |4Ys |5Ys |6Ys | 7Ys | 8Ys | 9Ys | 10 
= = = = = = = = = Ys= 


= Total of All Three Key Elements (3.4) 


Scoring for PQI 3: 


The number of positive records (all Ys for all three elements) where 
there is a crosswalk from developmental assessment to curriculum 


Percent of positive records (all Ys) (divide the number of positive 
records by 10 x 100%) %. Then based on the 
percentage, you can find the score of 1-4 as per the chart below. 


Circle the 


Appropriate Level 


INDICATOR 4): Opportunities for Staff and Families to Get to 
Know Each Other 


There should be activities both within the center as well as off site 
where staff and parents have opportunities to meet and greet each 
other. Communication with family members is documented and 
enables early childhood providers to assess the need for follow-up. 
Early childhood providers hold regular office hours when they are 
available to talk with family members either in person or by phone. 
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Family members are encouraged to lead the conversation and to 
raise any questions or concerns. 


How to Measure: 


Look for the following 3 examples in policies developed by the 
program and determine if they have been carried out with families. 
It will be necessary to interview staff to complete this indicator if 
you do not find the three examples in policies: 


1. The program provides communication, education, and 
informational materials & opportunities for families that 
are delivered in a way that meets their diverse needs. 

Y/N (4.1) 

2. The program communicates with families using different 
modes of communication, and at least one mode promotes 
two-way communication. Y/N (4.2) 

3. The program demonstrates respect and engages in ongoing 
two-way communication. The program respects each 
family’s strengths, choices, & goals for their children. Y/N 
___ (4.3) 


Scoring for PQI 4: 


Record the number of Yes’s (Y’s): (Range: 0 — 3) (Divide by 
3 x 100% = %). Then based on the percentage, you can find 
the score of 1-4 as per the chart below. 


Circle the 


Appropriate Level 


INDICATOR 5): Families Receive Information on Their Child’s 
Progress Regularly Using a Formal Mechanism 


Based upon Indicator #3 above, the information gleaned from the 
developmental assessments should be the focus of the report or 
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parent conference. Parental feedback about the assessment and 
how it compares to their experiences at home would be an 
excellent comparison point. All these interactions should be done 
in a culturally and linguistically appropriate way representing the 
parents being served. 


How to Measure: 


Look for the following four examples in policies developed by the 
program and determine if they have been carried out with families. 
Record the number of reports completed or parent conferences 
over the past year. It will be necessary to interview staff to 
complete this indicator if you cannot determine from records that 
the conferences or reports were completed. 


NOTE: The examples are mutually exclusive and are not additive; 
the first example is the highest scored, the third example the least 
scored. After 1-3 are determined, then do the last example. 


e 1) The program does have regularly scheduled (at least 
2xs/year) parent conferences in which the children’s 
developmental progress is discussed AND provides the 
family with a report of their child’s developmental 
progress. Y/N (5.1) (Score 3 points). If “Yes” then go 
to Number 4. If “No”, then go to numbers 2 and 3. 

e 2) The program has regularly scheduled (at least 2xs/year) 
parent conferences in which the children's developmental 
progress is discussed, but it does not provide a report to 
the parents on their child’s developmental progress. Y/N 
____ (5.2) (Score 2 points). 

e 3) If the program does not have regularly scheduled (at 
least 2xs/year) parent conferences, does it provide the 
family with a report of their child's developmental 
progress. Y/N (5.3) (Score 1 point). Go to Number 
4. 

e 4) All these interactions are done in a culturally and 
linguistically appropriate way representing the parents 
being served. Y/N (5.4) (Score 1 point) 


254 


Scoring for PQI5: 


Add up the total points based on the Ys; this will range from “O” to 
“4”, The only way a program can receive a “4”, is if a program has 
regularly scheduled parent conferences at least 2xs/year and 
provides the family with a report of their child’s progress; and it is 
done ina culturally and linguistically appropriate way. 


Record the number of points: (Range: 0 - 4) 


Total Score for Part 1 = 


PART 2 - OBSERVATIONS: 


For quality key indicators 6, 7 and 8, it is recommended that the 
licensing consultant refer to the appropriate Environmental Rating 
Scale (ERS) tool as a reference tool because these indicators are 
taken directly from these tools. It is also recommended that these 
be assessed/observed throughout the assessment and not just 
during key activity times. Please follow the specific instructions and 
examples as delineated below and in the appropriate ERS tool: 
ECERS (Items 12 and 13) or ITERS (item 12). These specific 
instructions and examples are provided within this tool for ease of 
administration and data collection. If there are several preschool 
aged classrooms randomly select one to do your observations. 


INDICATOR 6): Educators Encourage Children to 


Communicate (Preschool Class) 


Assessors will need to observe this item when they do their 
classroom observations. Initially you can ask educators or the 
director how children are encouraged to communicate but in order 
to gather reliable and valid information regarding this 
question/standard, it needs to be observed in the various 
interactions between staff and children. Things to look for would 
be more back and forth conversations rather than one-way 
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conversations where educators are telling children what to do. 
Look for opportunities where children can describe what they are 
doing, how they feel about what they are doing, and why they are 
doing particular activities. Educators expand upon children’s 
conversations. These opportunities can occur anywhere in the 
classroom or outside, such as in dramatic play, tabletop activities 
or on the playground. Materials should be present that encourage 
communication such as toy telephones, puppets, flannel boards, 
dolls and dramatic play props, small barns, fire stations, or 
dollhouses. These create a lot of conversation among children as 
they assume many different roles. Children also talk when there is 
an interested person who listens to them. The staff in a high- 
quality early childhood classroom will use both activities and 
materials to encourage growth in communication skills. 


How to Measure: 


Observe the classroom for a minimum of 15 minutes. Once 
completed, consider where the classroom falls based on the 
following scale; 

Score the classroom a 1 if the following occur: 

e No activities used by staff with children to encourage them 
to communicate, for example: nontalking about drawings, 
dictating stories, sharing ideas at circle time, finger plays, 
singing songs. Y/N (6.1) 

e Very few materials accessible that encourage children to 
communicate. Y/N (6.2) 

Score the classroom a 2 if the following occur (If the classroom 
does not have all 3 indicators but has 2 of the indicators then score 
this item 1+): 

e Some activities are used by staff w/children to encourage 


them to communicate. Y/N (6.3) 

e Some materials are accessible to encourage children to 
communicate. Y/N___ (6.4) 

e Communication activities are generally appropriate for the 
children in the group. Y/N (6.5) 
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Score the classroom a 3 if the following occur (If the classroom 
does not have both indicators but has one of the indicators then 
score this item 2+): 

e Communication activities take place during both free play 
and group times, for example: child dictates story about 
painting; small group discusses trip to store. Y/N 
(6.6) 

e Materials that encourage children to communicate are 
accessible in a variety of interest centers, for example: 
small figures and animals in block area; puppets and 
flannel board pieces in book area; toys for dramatic play 
outdoors or indoors. Y/N (6.7) 

Score the classroom a 4 if the following occur (If the classroom 
does not have both indicators but has one of the indicators then 
score this item 3+): 

e Staff balance listening and talking appropriately for age and 
abilities of children during communication activities, for 
example: leave time for children to respond; verbalize for 
child with limited communication skills. Y/N (6.9) 

e = Staff link children’s spoken communication with written 
language, for example: write down what children dictate & 
read it back to them; help them write notes to parents. 
Y/N (6.10) 

Scoring for PQI 6: 

Total up the number of “Y’s” and record the appropriate level. In 
order for a classroom to receive a particular score, all “Y’s” must be 
checked for the appropriate level (1 - 4) from above or partial credit 
given in order to obtain a “+”. If there is a “+” please also mark it in 
the box. 


” 


Circle the 


Appropriate Level 
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INDICATOR 7): Infant Toddler Observation (if applicable) 
(Infant Classroom) 


NOTE: If there is an infant, toddler or combined infant/toddler 
classroom that needs to be assessed, then use the following ITERS 
item directly from the ITERS Tool (Item 12), if there is not an infant 
toddler classroom, then skip to Indicator 8. 

Conversations and questions should be used with all children, even 
young infants. Conversations using verbal and nonverbal turn- 
taking should be considered when scoring. Most conversations and 
questions initiated by infants will be nonverbal, such as widening of 
baby’s eyes or waving arms and legs. Observe staff response to 
such nonverbal communication. For infants and toddlers, the 
responsibility for starting most conversations and asking questions 
belongs to the staff. As children become more able to initiate 
communication, staff should modify their approach in order to 
allow children to take on a greater role in initiating conversations 
and asking questions. Staff should provide answers to questions 
used by children if children cannot answer, and as children become 
more able to respond, questions should start to include those that 
the child can answer. If there was not an infant classroom, skip this 
Indicator and please note that here and on the summary score 
sheet by marking N/A: 

How to Measure: 

Observe the classroom for a minimum of 15 minutes. Once 
completed, consider where the classroom falls based on the 
following scale; 

Score the classroom a 1 if the following occurs: 

e Staff never initiate turn-taking conversations with children, 
for example: rarely encourage baby to babble back; simple 
back and forth exchanges with verbal children never 
observed. Y/N (7.1) 

e Staff questions are often not appropriate for children, or 
no questions are asked, for example: too difficult to 
answer; carry a negative message. Y/N (7.2) 
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Staff respond negatively when children can’t answer 
questions, for example: “You should know this”; “You did 
not listen”. Y/N (7.3) 


Score the classroom a 2 if the following occurs (If the classroom 
does not have all 3 indicators but has 2 of the indicators then score 
this item 1+): 


Staff sometimes initiate conversations with children, for 
example: babble back and forth with baby; copy baby’s 
sounds; respond to baby’s crying with verbal response; 
have short back and forth toddler interactions. Y/N 

(7.4) 

Staff sometimes ask children appropriate questions and 
wait for the child to respond, for example: ask baby if she 
likes toy and pay attention as baby smiles; ask toddler what 
he is eating and wait for him to think of word. Y/N 

(7.5) 

Staff respond neutrally or positively to children who can’t 
answer questions. Questions asked are sometimes 
meaningful to children, for example: child responds with 
interest; does not ignore staff questions. Y/N (7.6) 


Score the classroom a 3 if the following occurs (If the classroom 
does not have all 4 indicators but has 2 or more of the indicators 
then score this item 2+): 


Staff initiate engaging conversations with children 
throughout the observation, for example: show 
enthusiasm; use tone that attracts child’s attention. Y/N 
_» (7.7) 

Staff often personalize questions and/or conversations for 
individual children, for example: talk about children’s 
families, preferences, interests; what they are playing with; 
what they did over weekend; child’s mood; use child’s 
name. Y/N (7.8) 

Staff often pay attention to children’s questions, verbal or 
nonverbal, and answer in a satisfying manner for the child. 
Y/N (7.9) 

Staff ask questions in which children show interest in 
answering, for example: make the questions funny or 
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mysterious; use attractive tone; meaningful and not too 
difficult to answer. Y/N (7.10) 
Score the classroom a 4 if the following occurs (If the classroom 
does not have both indicators but has one of the indicators then 
score this item 3+): 

e =Staff frequently have turn taking conversations with 
children throughout the observations. Many appropriate 
questions are used throughout the observation, during 
both play and routines. Y/N (7.11) 

e = Staff ask children appropriate questions, wait a reasonable 
time for child response, and then answer if needed, for 
example: “Are you hungry? ... Yes, you are!”; “Where’s 
the ball? ... These itis! You found the ball”. Y/N 
(7.12) 

Scoring for PQI 7: 

Total up the number of “Y’s” and record the appropriate level. For 

a classroom to receive a particular score, all “Y’s” must be checked 

for the appropriate level (1 - 4) from above or partial credit given in 
order to obtain a “+”. 


Circle the 


Appropriate Level 


INDICATOR 8): Educators Use Language to Develop 
Reasoning Skills (Preschool) 


Assessors will need to observe very carefully as this standard can 
be difficult to determine because it is tying language and cognition 
together. Again, this opportunity can occur in any setting in or out 
of the classroom because it is the basis for problem solving through 
the use of language. Also look for educators redirecting children’s 
conversations when appropriate. Staff should use language to talk 
about logical relationships using materials that stimulate reasoning. 
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Through the use of materials, staff can demonstrate concepts such 
as same/different, classifying, sequencing, one-to-one 
correspondence, spatial relationships, and cause and effect. 


How to Measure: 

Observe the classroom for a minimum of 15 minutes. Once 
completed, consider where the classroom falls based on the 
following scale; 

Score the classroom a 1 if the following occur: 


Staff do not talk with children about logical relationships, 
for example: ignore children's questions and curiosity 
about why things happen, do not call attention to 
sequence of daily events, differences and similarity in 
number, size, shape, cause and effect. Y/N (8.1) 
Concepts are introduced inappropriately, for example: 
concepts too difficult for age and abilities of children, 
inappropriate teaching methods used such as worksheets 
without any concrete experiences; teacher gives answers 
w/o helping children to figure things out. Y/N (8.2) 


Score the classroom a 2 if the following occur (If the classroom 
does not have both indicators but has one of the indicators then 
score this item 1+): 


Staff sometimes talk about logical relationships or 
concepts, e.g.: explain that outside time comes after 
snacks, point out differences in sizes of blocks children use. 
Y/N (8.3) 

Some concepts are introduced appropriately for ages and 
abilities of children in group, using words and experiences, 
for example: guide children with questions and words to 
sort big and little blocks or to figure out why ice melts. Y/N 
____ (8.4) 


Score the classroom a 3 if the following occur (If the classroom 
does not have both indicators but has one of the indicators then 
score this item 2+): 


Staff talk about logical relationships while children play 
with materials that stimulate reasoning, for example: 
sequence cards, same/different games, size and shape 
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toys, sorting games, numbers and math games. Y/N 
(8.5) 

e Children are encouraged to talk through or explain their 
reasoning when solving problems, for example: why they 
sorted objects into different groups, in what way two 
pictures are the same or different. Y/N (8.6) 

Score the classroom a 4 if the following occur (If the classroom 
does not have both indicators but has one of the indicators then 
score this item 3+): 

e Staff encourage children to reason throughout the day, 
using actual events and experiences as a basis for concept 
development, e.g.: children learn sequence by talking 
about their experiences in the daily routine or recalling the 
sequence of a cooking project. Y/N (8.7) 

e Concepts are introduced based upon children's interests or 
needs to solve problems, for example: talk children 
through balancing a tall block building, help children figure 
out how many spoons are needed to set a table. Y/N 
(8.8) 

Scoring for PQI 8: 

Total up the number of “Y’s” and record the appropriate level. In 
order for a classroom to receive a particular score, all “Y’s” must be 
checked for the appropriate level (1 - 4) from above or partial credit 
given in order to obtain a “+”. 


Circle the 


Appropriate Level 


For quality key indicators 9 and 10 it is recommended that these be 
assessed/observed throughout the observation period and not just 
during key activity times. These two quality key indicators should 
be observed in two-minute blocks over ten sequences for a total of 
20 minutes. These two items should also be used with each age 
group you are assessing. 
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INDICATOR 9): Educators Listen Attentively When Children 
Speak 

This quality indicator focuses on the early childhood educator(s) 
looking directly at the children with nods, rephrases their 
comments, engages in conversations. Children should have the 
undivided attention of the specific educator they are addressing. 
Educators should not be looking away or pre-occupied with others. 
They should be at the child’s level making eye contact. The intent is 
to observe all children and educators in the room. 


How to Measure: 


Do this in timed 2-minute observations recording each time you 
observe this occurring. Record at least 10 different observation 
periods. These do not need to be consecutive in order to fully 
observe classrooms and educators. Please use the following scale 
to assess your recordings: Likert Scale (1-4) where 1 = Never/Not at 
All; 2 = Somewhat/Few Instances; 3 = Quite a Bit/Many Instances; 4 
= Very Much/Consistently): 

Make the actual recordings using the Likert Scale (1-4) above for 
each individual observation and record in each cell below. 

10 Observations: 


Scoring for PQI 9: 


Once all the observations are made, add up the results from the 
Likert Scale (1-4) and record the total number here: 
(Range: 10 - 40)(Divide this result by 10) = 
(1-4)(Round upward or downward to the whole 
number (3.7 = 4; 2.2 = 2)). 
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Circle the 


Appropriate Level 


INDICATOR 10): Educators Speak Warmly to Children 


This quality indicator focuses on the early childhood educator(s) 
always engaging in a caring voice and body language with every 
child. Educators do not use harsh language or commands in 
speaking to children, but rather again are on the child’s level 
making eye contact. Think of the way Fred Rogers would engage 
his audience where you always felt you were the most important 
person in the world when he talked to the TV. 


How to Measure: 

Do this in timed 2-minute observations recording each time you 
observe this occurring. Record at least 10 different observation 
periods. Please use the following scale to make your recordings: 
(This item is on a Likert Scale (1-4) where 1 = Never/Not at All; 2 = 
Somewhat/Few Instances; 3 = Quite a Bit/Many Instances; 4 = Very 
Much/Consistently): 

Make the actual recordings using the Likert Scale (1-4) above for 
each individual observation and record in each cell below. 

10 Observations: 


Scoring for PQI 10: 


Once all the observations are made, add up the results from the 
Likert Scale (1-4) and record the total number here: 
(Range: 10 - 40) (Divide this result by 10) = 
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(1-4). (Round upward or downward to the whole 
number (3.7 = 4; 2.2 = 2)). 


Circle the 


Appropriate Level 
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SKECPQI Scoring Protocol 


Standardized Actual Scores 


Scores 


Mixed Age: 36+ Mixed Age: 
High Quality Preschool: 32+ 
Infant-Toddler: 28+ § Preschool: 
Infant-Toddler: 


Mixed Age: 30-35 J Mixed Age: 
High - Mid Quality Preschool: 26 - 31 
Infant-Toddler: 22 - | Preschool: 
27 
InfantToddler: 


Mixed Age: 20-29 J Mixed Age: 
Mid — Low Quality Preschool: 16 - 25 
Infant-Toddler: 12 - | Preschool: 
21 


Infant-Toddler: 


Mixed Ages: 19 or § Mixed Age: 
Low Quality less 
Preschool: 15 or Preschool: 
less 
Infant-Toddler: 11 J Infant-Toddler: 
or less 
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Note: 


Members of the Original Saskatchewan Program Quality Work 
Group are the following: 

Ministry of Education: Kim Taylor, Derek Pardy, Cindy Jeanes, 
Tanya Mengel, Samantha Ecarnot, Karen Heinrichs, Michelle 
Vellenoweth, Kristin Jarvis, and NARA Consultant: Rick Fiene. 


Additional Information regarding the psychometrics of the tool contact: 
Richard Fiene, Ph.D., Research Psychologist, Research Institute for Key 
Indicators & Penn State University. RFiene@RIKInstitute.com or 
RFiene@NARALicensing.org 


10/2020; 4/2021; 1/2023; 2/2023; 3/2023 versions 


After completing your observations, reviewing all documentation, 
and interviewing staff, when necessary, please transfer all your 
results to the Summary Table below. If there was not an infant 


classroom, please note here, no infant classroom: . If there 
was not a toddler classroom, please note here, no toddler 
classroom: . If there was not a preschool classroom, please 


note here, no preschool classroom: 
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Professional 
Development 
The Environment 
Curriculum and 


Assessment 


Family Engagement | 


Family Engagement 


Communication 
(Preschool) 


Infant Classroom 


Reasoning Skills 
(Preschool) 
Listen Attentively 
Speak Warmly 
Notes: 
Use ITERS if: (Infants) (B-1yr) 


Use ITERS if: (Toddlers) (1yr-2yr) 
Use ECERS if: (Preschoolers) (3yr+) 
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SKECPQI/Infant (administer QKI items 1-5, 7, 9-10) (Scores 8-32) 


SKECPQI/Toddler or Preschool (administer QKI items 1-5, 7, 9-10) (Scores 
8-32) or (administer QKI items 1-6, 8-10) (Scores 9-36). Mixed age group 
(administer QKI items 1-10) (Scores 10-40) 


SKECPQI/Preschool (administer QKI items 1-6, 8-10) (Scores 9-36) 


All the above 10 quality indicators (SKECPQI) have been taken from 
other sources having been identified in Quality Indicator Studies 
conducted by Dr Richard Fiene from 1980 — 2020. Please refer to the 
source documents for details on their creation: ECERS, ITERS, 
QRIS/INQUIRE, CIS/Arnett, NAEYC, SASKATCHEWAN PLAY & 
EXPLORATION. For additional information, reports, and publications 
related to these studies, please go to 


On July 27, 2023 this Early Childhood Quality Indicators Project 
was recognized by the World Forum Foundation as a Child Impact 
Initiative. 
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SKECPQI: SASKATCHEWAN EARLY CHILDHOOD PROGRAM 


QUALITY INDICATORS CHART/GRAPH 


Scores 
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Quality Key 


Indicators (QKI) 


QKI Scores 


Elements/Items 


Data Collection 


Record Review 


Policy, Records, 
Interviews 


Policy, Records, 
Interviews 


Policy, Records, 
Interviews 


Policy, Records, 
Interviews 


Observation 


Observation 


Observation 


Observation 


Observation 


Potential Score = 
78 
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Actual Score 
Obtained = 


The Uncertainty-Certainty Matrix for Licensing Decision 
Making, Validation, Reliability, and Differential 
Monitoring Studies 


This research abstract will take the Confusion Matrix which is 
a well-known metric in the decision-making research literature 
and refocus it on regulatory science within the context of the 
definition of regulatory compliance and licensing measurement. 
It will also deal with the policy implications of this particular 
metric. In this abstract, it is proposed that the Uncertainty- 
Certainty Matrix (UCM) is a fundamental building block to 
licensing decision making. The 2 x 2 matrix has been written 
about in several posts in this blog and is the center piece for 
determining key indicator rules, but it is also a core conceptual 
framework in licensing measurement and ultimately in program 
monitoring and reviews. 


The reason for selecting this matrix is the nature of licensing 
data, it is binary or nominal in measurement. Either a 
rule/regulation is in compliance or out of compliance. Presently 
most jurisdictions deal with regulatory compliance 
measurement in this nominal level or binary level. There is to 
be no gray area, this is a clear distinction in making a licensing 
decision about regulatory compliance. The UCM also takes the 
concept of Inter-Rater Reliability (IRR) a step further in 
introducing an uncertainty dimension that is very important in 
licensing decision making which is not as critical when 
calculating IRR. It is moving from an individual metric to a 
group metric (See Figures 1 & 2) involving regulatory 
compliance with rules. The key pieces to the UCM are the 
following: the decision (D) regarding regulatory compliance 
and actual state (S) of regulatory compliance. Plus (+) = In- 
compliance or Minus (-) = Out of compliance. So, let’s build 
the matrix: 
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Table 1: Uncertainty-Certainty Matrix (UCM) Logic Model 


UCM Matrix 
Logic 


Decision (D) 
Regarding 


Regulatory 
Compliance 


(+) In 
Compliance 


(-) Not In 
Compliance 


Actual State 
(S) of 


(+) In 
Compliance 


Agreement 


Disagreement 


Compliance 


(-) Not In 


Disagreement 


Agreement 


Compliance 


The above UCM matrix demonstrates when agreement and 
disagreement occur which establishes a level of certainty 
(Agreement Cells) or uncertainty (Disagreement Cells). In a 
perfect world, there would only be agreements and no 
disagreements between the decisions made about regulatory 
compliance and the actual state of regulatory compliance. But 
from experience, this is not the case based upon reliability 
testing done in the licensing research field in which a decision 
is made regarding regulatory compliance with a specific rule or 
regulation and then that is verified by a second observer who 
generally is considered the measurement standard. 


Disagreements raise concerns in general, but the disagreements 
are of two types: false positives and false negatives. A false 
positive is when a decision is made that a rule/regulation is out 
of compliance when it is in compliance. Not a good thing but 
its twin disagreement is worse where with false negatives it is 
decided that a rule/regulation is in compliance when it is out of 
compliance. False negatives need to be avoided because they 
place clients at extreme risk, more so than a false positive. False 
positives should also be avoided but it is more important to deal 
with the false negatives first before addressing the false 
positives. 
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Let’s look at this from a mathematical point of view in the 
following matrix. In order to better understand the above 
relationships and determine when ameliorative action needs to 
occur to shore up the differences between the agreements and 
disagreements, it is easier to do this mathematically than trying 
to eyeball it. 


Table 2: Uncertainty-Certainty Matrix (UCM) Math Model 


UCM Decision (D) Regulatory 
Matrix Regarding Compliance 
Math 
Model 


(+) In (-) Not In 
Compliance Compliance 


Actual State | (+) In 

(S) Compliance 
Of (-) Not In 
Compliance | Compliance 
Totals 


Formulae based upon above: Agreements = (A)(D); 
Disagreements = (B)(C); Randomness = sqrt ((W)(X)(Y)(Z)) 


UCM Coefficient = ((A)(D)) - ((B)(C)) / sqrt ((W)(X)CY)(Z)) 
in which a coefficient closer to 1 indicates agreement (certainty) 
and a coefficient closer to -—1l indicates disagreement 
(uncertainty). A coefficient closer to 0 indicates randomness. 
Obviously, we want to see (A)(D) being predominant and very 
little in (B)(C) which are false positives and negatives where 
decisions and the actual state of regulatory compliance are not 
matching. If (WXYZ) is predominant then there is just 
randomness in the data. Also, not an intended result. 


The reason for even suggesting this matrix is the high level of 
dissatisfaction with the levels of reliability in the results of 
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program monitoring reviews as suggested earlier. If it were not 
so high, it would not be an issue; but with it being so high the 
field of licensing needs to take a proactive role in determining 
the best possible way to deal with increasing inter-rater 
reliability among licensing inspectors. Hopefully, this 
organizational schema via the UCM Matrix will help to think 
through this process related to licensing measurement and 
monitoring systems. 


UCM = ((A)(D)) - ((B)(C)) / sqrt ((W)(X)(Y)(Z)) 


The above formula provides a means to calculate when action 
needs to be taken based upon the respective UCM coefficients. 
A UCM coefficient from +.25 to +1.00 is in the acceptable 
range; +.24 to -.24 is due to randomness and needs to be 
addressed with additional inter-rater reliability training; -.25 to 
—1.00 indicates a severe disagreement problem that needs to be 
addressed both in reliability training and a full review of the 
targeted rules/regulations to determine if the specific rule needs 
additional clarification. 


Table 3: Uncertainty-Certainty Matrix (UCM) Licensing 
Decision Coefficient Ranges 


UCM Licensing Decision 
Coefficient 
+.25 to Acceptable, No Action Needed, In or Out of 
+1.00 Regulatory Compliance Verified through mostly 
Agreements. (Generally, 90% of cases) 

+.24 to -.24 | Random, Agreements + Disagreements, Needs 
Reliability Training. (Generally, 5% of cases) 

-.25 to — Unacceptable, Mostly Disagreements, Needs 

1.00 Training & Rule/Regulation Revision. (Generally, 
5% of cases) 
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Figure 1: Kappa Coefficient 


Observed agreement 


Va 
ee eee 
1 =p 
/ 


Expected agreement if 
random judgment 


Figure 2: Uncertainty-Certainty Coefficient 


ad — be 


J(a + b)(e+d)(a+c)(b+d) 


p= 


Let’s provide an example of how this could work. A 
standard/rule/regulation that is common is the following: 


Do all caregivers/teachers and children wash their hands often, 
especially before eating and after using the bathroom or 
changing diapers? 


This is obviously an observation item where the licensing staff 
would observe in a sample of classrooms in a child care center 
for a set period of time. During their observations, there were 
several opportunities where the necessary behavior was 
required, and the staff complied with the rule and washed their 
hands. So, on the surface this specific rule was in compliance 
and there would appear to be full compliance with this rule 
based upon the observation. 
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A second scenario is where the observation is made, and the 
licensing staff observes the child care staff not washing their 
hands on several occasions. Then this specific rule would be 
out of compliance, and it would be duly noted by the licensing 
staff. These two scenarios establish a certain level of certainty 
during this observation session. However, there are other 
outcomes, for example, possibly one of the classrooms that was 
not observed had the opposite finding than what was observed 
in these particular classrooms. If data were being aggregated 
and a specific percentage was to be used the final decision about 
this rule could be different. Now we are getting into the 
uncertainty cells of the matrix where a false positive or negative 
could be the result. The licensing staff records the rule as being 
in compliance when in reality it is not = false negative or the 
rule is recorded as being out of compliance when in reality it is 
in compliance = false positive. 


Another example which involves either Random Clinical Trials 
(RCT) or the use of abbreviated inspections (AJ) and the results 
from these two interventions. The decision making in both 
RCT and AT is basically the same. We want to make sure that 
the results match reality. Every time an abbreviated review is 
done the following four regulatory compliance results should 
occur based upon the UCM matrix: 1) no additional random 
non-compliance is found; 2) there are no false negatives 
(abbreviated review finds no non-compliance but in reality 
there is); 3) when there is non-compliance found in abbreviated 
inspections, other related non-compliance is found; and 4) lastly 
the level of false positives (abbreviated review finds non- 
compliance but in reality there are no other related non- 
compliances) is kept to a minimum. This last result based upon 
copious research is that it is difficult to obtain but as the 
regulatory science moves forward hopefully this will become 
more manageable. 
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Hopefully these above examples provided some context for 
how the Uncertainty-Certainty Matrix (UCM) can be used in 
making specific licensing decisions based upon the regulatory 
compliance results. 


Uncertainty-Certainty Matrix for Validation and 
Reliability Studies 


The purpose of this part of this research abstract is to explore 
the possibility of utilizing the Uncertainty-Certainty Matrix 
(UCM) in validation and reliability studies in licensing decision 
making. The UCM has been proposed for use in licensing 
decision making but this would be an extension of this thinking 
to studies that involve validating licensing decisions such as 
when key indicators are used in comparison with 
comprehensive reviews of rules, and in reliability studies to 
determine individual inspector bias in regulatory compliance. 


The basic premise of the UCM is that individual decision- 
making matches reality. When it comes to regulatory 
compliance decision making a 2 x 2 matrix can be drawn with 
the possible outcomes as is indicated in the following table: 


Table 4 


UCM Matrix Decision Regarding | Regulatory 
Logic Compliance 
(+) In Compliance (-) Not In Compliance 


Actual State of (+) In Compliance Agreement (++) Disagreement (+-) 


Compliance (-) Not In Disagreement (-+) Agreement (--) 
Compliance 


In using this table, the hope is that the decision regarding 
regulatory compliance matches the actual state of compliance 
where the coefficient is as close to +1.0 as possible, in other 
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words, perfect agreement. So, the agreement cells are heavily 
weighted. We do not want to see all the cells, both agreement 
and disagreement cells, equally weighted. That would indicate 
a random response rate and a coefficient close to 0.0. 


But there is another possibility which involves bias on the part 
of the licensing inspector in which they have certain biases or 
tendencies when it comes to making regulatory compliance 
decisions about individual rules. So, it is possible that decisions 
made regarding regulatory compliance could be either overall 
(+) positive In-Compliance or (-) negative Not-In-Compliance 
when in reality the actual state of compliance is more random. 


When this occurs, the coefficient falls off the range category 
and is not between 0 and +/-1.0 because there is no variance 
detected in the data. It is always biased either positively or 
negatively. 


The UCM can be used for both reliability and validity testing as 
suggested in the above. Just look for different results. For 
validity, false positives and negatives should either be 
eliminated or reduced as well as possible and the remaining 
results should show the typical diagonal pattern as indicated by 
the agreement cells. 


For reliability, the same pattern should be observed as in the 
validity testing above but there is an additional test in which 
bias is tested for. Bias will be ascertained if the patterns in the 
results indicate a horizontal or vertical pattern in the data with 
little or no diagonal indication. Bias can be found at the 
individual inspector level as well as at the standard level or the 
actual state of compliance. 
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In both reliability and validity testing, random results in which 
each of the cells are equally filled are not a desirable result 
either. The following tables 5-10 depict the above relationships 


with results highlighted in red: 


Table 5 


Valid & Reliable 
Results 


(+) In Compliance 


(-) Not In Compliance 


(+) In Compliance 


Agreement (++) 


Disagreement (+-) 


(-) Not In Compliance 


Table 6 


Random Results 


Disagreement (-+) 


(+) In Compliance 


Agreement (--) 


(-) Not In Compliance 


(+) In Compliance 


Agreement (++) 


Disagreement (+-) 


(-) Not In Compliance 


Disagreement (-+) 


Agreement (--) 


Table 7 


Positive Bias Results 
Individual 


(+) In Compliance 


(-) Not In Compliance 


(+) In Compliance 


Agreement (++) 


Disagreement (+-) 


(-) Not In Compliance 


Table 8 


Negative Bias Results 
Individual 


Disagreement (-+) 


(+) In Compliance 


Agreement (--) 


(-) Not In Compliance 


(+) In Compliance 


Agreement (++) 


Disagreement (+-) 


(-) Not In Compliance 


Disagreement (-+) 
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Agreement (--) 


Table 9 


Positive Bias Results (+) In Compliance (-) Not In Compliance 
Standard 
(+) In Compliance Agreement (++) Disagreement (+-) 


(-) Not In Compliance Disagreement (-+) Agreement (--) 


Table 10 


Negative Bias Results (+) In Compliance (-) Not In Compliance 
Standard 
(+) In Compliance Agreement (++) Disagreement (+-) 


(-) Not In Compliance Disagreement (-+) Agreement (--) 


Tables 5 — 10 demonstrate the different results based upon 
individual response rates when making regulatory compliance 
decisions about rules. Table 5 is what needs to be attained and 
tables 6 — 10 need to be avoided. Only in table 5 are false 
negatives and positives eliminated or avoided. In tables 6 — 10, 
false negatives and/or false positives are introduced which is 
not desirable when making validity or reliability decisions. 


Table 6 results clearly indicate that a great deal of randomness 
has been introduced in the regulatory compliance decision 
making in which the individual licensing inspector decisions do 
not match reality. Tables 7 and 8, demonstrate bias in the 
decision-making process either positively (inspector always 
indicates in compliance) or negatively (inspector always 
indicates out of compliance). It is also possible that the 
standard being used has bias built into it, this is less likely but 
is still a possibility. The results in Tables 9 and 10 demonstrate 
where this could happen. 
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All these scenarios need to be avoided and should be monitored 
by agency staff to determine if there are patterns in how 
facilities are being monitored. 


Uncertainty-Certainty Matrix for Differential Monitoring 
Studies 


The purpose of this part of the research abstract is to explore the 
possibility of utilizing the Uncertainty-Certainty Matrix (UCM) 
not only in validation and reliability studies in licensing 
decision making but also with differential monitoring studies. 
The UCM has been proposed for use in licensing decision 
making but this would be an extension of this thinking to studies 
that involve validating licensing decisions such as when key 
indicators are used in comparison with comprehensive reviews 
of rules, and in the development of risk rules as part of the risk 
assessment methodology. This new Differential Monitoring 
2x2 Matrix can also be used to depict the relationship between 
full and substantial regulatory compliance and the nature of 
rulemaking. 


The basic premise of the DMM: Differential Monitoring Matrix 
is similar to the original thinking with the UCM but there are 
some changes in the formatting of the various cells in the matrix 
(see Table 11). When it comes to regulatory compliance 
decision making a 2 x 2 matrix can be drawn with the possible 
outcomes as is indicated in Table 11 where each individual rule 
is either in (+) or out (-) of compliance. Also, there is the 
introduction of a high regulatory compliant group (+) and a low 
regulatory compliant group (-) which is different from the 
original UCM. 
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Table 11 


DMM Matrix High Group (+) Low Group (-) 
(+) Rule is In (++) (+-) 


Compliance 
(-) Rule is Not In (-+) (--) 
Compliance 


By utilizing the format of Table 11, several key components of 
differential monitoring can be highlighted, such as key 
indicators and risk assessment rules, as well as the relationship 
between full and substantial regulatory compliance. 


Regulatory compliance is grouped into a high group (+), 
generally this means that there is either full or substantial 
regulatory compliance with all rules. The low group (-) usually 
has 10 or more regulatory compliance violations. Individual 
rules being in (+) or out (-) of regulatory compliance is self- 
explanatory. 


Tables 12-18 below will demonstrate the following 
relationships: 


Table 12 depicts the key indicator relationship between 
individual rules and the high/low groups as indicated in red. In 
this table, the individual rule is in compliance with the high 
group and is out of compliance with the low group. This result 
occurs on a very general basis and should have a .50 coefficient 
or higher with a p value of less than .0001. 


Table 13 depicts what most rules look like in the 2x2 DMM. 

Most rules are always in full compliance since they are 

standards for basic health and safety for individuals. This is 

especially the case with rules that have been weighted as high- 
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risk rules. Generally, one never sees non-compliance with these 
rules. There will be a substantial number of false positives (+- 
) found with high-risk rules but that is a good thing. 


Table 14 depicts what happens when full compliance is used as 
the only criterion for the high group. Notice that the cell right 
below (++) is eliminated (-+). This is highly recommended 
since it eliminates false negatives (-+) from occurring in the 
high group. As will be seen in Table 5, when substantial 
compliance is used as part of the high group sorting, false 
negatives are re-introduced. If possible, this should be avoided, 
however in some cases because of the regulatory compliance 
data distribution it is not always possible where not enough full 
compliant programs are present. 


Table 15 depicts what occurs when substantial compliance is 
used as part of determining the high group. False negatives can 
be reintroduced into the matrix which needs to be either 
eliminated or reduced as best as possible. If substantial 
compliance needs to be used in determining the high group, 
then there is a mathematical adjustment that can be made which 
will impact the equation and essentially eliminate false 
negatives mathematically (see the research note at the end of 
this research abstract). 


Table 16 depicts what happens if the individual rule is 
particularly difficult to comply with. Both the high performers 
as well as the low performers are out of compliance with the 
rule. 


Table 17 depicts a situation where the programs are 
predominantly in a low group with few at full or substantial 
regulatory compliance which is indicative of poor performing 
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programs. Very honestly, this is generally not seen in the 
research literature, but it is a possibility and one to be in tune 
with. 


Table 18 depicts a terrible individual rule which predicts just 
the opposite of what we are trying to do with programs. 
Obviously, this rule would need to be rewritten so that it fits 
with the essence of regulatory compliance in helping to protect 
individuals. 


The following tables 12-18 will depict the above relationships 
with results highlighted in red: 


Table 12 


Key Indicators High Group (+) Low Group (-) 
(+) Rule is In (++) 


Compliance 
(-) Rule is Not In (-+) (--) 
Compliance 


Table 13 


Risk Rules High Group (+) Low Group (-) 
(+) Rule is In (++) (+-) 

Compliance 
(-) Rule is Not In (-+) (--) 
Compliance 


Table 14 


Full Compliance High Group (+) Low Group (-) 
(+) Rule is In (++) 
Compliance 
(-) Rule is Not In (--) 
Compliance 
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Table 15 


Substantial 
Compliance 


High Group (+) 


Low Group (-) 


(+) Rule is In 
Compliance 


(if) 


(-) Rule is Not In 
Compliance 


Table 16 


Very Difficult Rule 


=) 


High Group (+) 


Low Group (-) 


(+) Rule is In 
Compliance 


(++) 


(-) Rule is Not In 
Compliance 
Table 17 


Poor Performing 
Programs 


(-+) 


High Group (+) 


>) 


Low Group (-) 


(+) Rule is In 
Compliance 


Ca) 


(-) Rule is Not In 
Compliance 


Table 18 


Terrible Rule 


(+) 


High Group (+) 


Low Group (-) 


(+) Rule is In 


Compliance 


(++) 


(-) Rule is Not In 
Compliance 


in) 


t) 


Tables 12 — 18 demonstrate the different results based on the 

relationship between individual regulatory compliance and if a 

program is either a high performer or a low performer. These 
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tables are provided as guidance for understanding the essence 
of differential monitoring and regulatory compliance which has 
various nuances when it comes to data distributions. This 
research abstract hopefully can be used as a guide in 
determining from a data utilization point of view how to make 
important regulatory compliance policy decisions, such as: 
which rules are excellent key indicator rules, which are 
performing as high risk rules, importance of full compliance, 
what to do when substantial compliance needs to be employed, 
are there difficult rules to comply with, how well are our 
programs performing, and do we have less than optimal rules 
that are in need of revision. 


Research Note: 


Over the past decade in doing research on the Regulatory 
Compliance Key Indicator Metric (RCKIm) it has become very 
clear that false negatives needed to be controlled for because of 
their potential to increase morbidity and mortality. When 
dealing with regulatory compliance and full compliance as the 
threshold for the high grouping variable in the 2 x 2 Regulatory 
Compliance Key Indicator Matrix (RCKIm) (see matrix below 
in Table 19), false negatives could be either eliminated or 
reduced to the point of no concern. 


However, if substantial compliance rather than full 
compliance is used as the threshold for the high grouping 
variable in the 2 x 2 Regulatory Compliance Key Indicator 
Matrix (RCKIm) this becomes a problem again. There is the 
need to introduce a weighting factor. In utilizing the RCKIm, 
the following equation/algorithm is used to produce the Fiene 
Coefficient (FC): 
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FC = ((A)(D)) — (B)(C)) / sqrt (WXYZ) 


This RCKIm needs to be revised/updated to the following to 
consider the need to again eliminate false negatives being 
generated by the results of the equation/algorithm; this can be 
accomplished by cubing C: 


FC* = ((A)(D)) — ((B)(C%3)) / sqrt(WX YZ) 


By this simple adjustment to cube (C = False Negatives) it will 
basically eliminate the use of any results in which a false 
negative occurs when substantial compliance is determined. 
The table below (Table 19) displays the variables of the 
Regulatory Compliance Key Indicator Matrix (RCKIm). 


Table 19: High RC 
RCKIm RC Low 
Group Group 
KI In A B Y 
Compliance 
KI C3 D Z 
Violations 
Totals WwW x 


In the above examples, FC can be used when the High RC 
Group is at full regulatory compliance, but FC* needs to be 
used when the High RC Group is including substantial as well 
as full regulatory compliance. By using both 
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equations/algorithms, it better deals with the results of the 
Regulatory Compliance Theory of Diminishing Returns. 


The results should clearly show that only positive (+) 
coefficients will become Regulatory Compliance Key 
Indicators versus those rules that do not show any 
relationship to overall regulatory compliance (0), but now the 
negative (-) coefficients will more clearly show when any 
false negatives appear and clearly not include them as 
Regulatory Compliance Key Indicators (see Figure 2). This 
is a Major improvement in the Regulatory Compliance Key 
Indicator methodology which clearly demonstrates the 
differences in the results. It provides a gateway in regulatory 
compliance data distributions where substantial regulatory 
compliance is heavily present while full regulatory 
compliance is not. This could become a problem as the 
regulatory science field moves forward with the use of the 
Regulatory Compliance Theory of Diminishing Returns. 
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Regulatory Compliance Scale Weighted (RCSw) 


The Regulatory Compliance Scale (RCS) has been proposed as 
a more effective and efficient means of measuring regulatory 
compliance rather than adding up the number of regulatory 
violations for a specific program. It improves upon the 
measurement of rules by moving from a nominal to an ordinal 
level. However, it does have a limitation in that in its present 
form it is based upon violation frequency data that are not 
weighted according to the specific risk factor of each rule. This 
brief technical research abstract will attempt to correct this 
deficiency by moving the scaling of the RCS by considering 
weighting and relative weighting of rules based upon the risk of 
rule non-compliance for clients. 


In the following table, three models for RCS are displayed 
based upon violation data, weights, and relative weights. 
Violation (Violations Model) data have no weights or another 
way of thinking about it as all rules have the same weight, a 
weight of one (1); weighting (Weights Model) is done on a 
Likert Scale generally using 1-9 scaling, while relative 
weighting (RWeights Model) is a new approach utilizing the 
Fibonacci Sequence which uses a 1-100 scaling. 


Table Comparing the Regulatory Compliance Scale Models 


Models 
Scale Compliance Risk Violations Weights RWeights 
Level 
7 Full None 0 0 0 
5 Substantial Low 1-3 1-3 1-3 
3 Medium Medium 4-9 4-6 4-19 
1 Low High 10+ 7+ 20+ 
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With these new models proposed, it tremendously improves 
upon the Regulatory Compliance Scale (RCS) by considering 
the relative risk of each rule which was not the case in the 
original RCS. The new RCSw versions do this through the 
weights or relative weights applied to each rule based upon an 
equal Weighting Schema or a Fibonacci Sequence. The 
addition of the weights to the RCS should make for a more 
accurate determination of regulatory compliance. However, 
these new models have not been tested empirically as of this 
writing. It is planned in the upcoming year to test them out side 
by side and compare them to the original RCS Violation Model. 
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Uniform, Differential, and Integrated Program Monitoring 


This technical section demonstrates the key similarities and 
differences amongst uniform, differential, and integrated 
monitoring. The similarities and differences will be depicted in 
the following table. The table depicts how each monitoring 
approach addresses the specific key element presented. 
Explanations are provided after the summary table. It builds off 
several other papers* that dealt with regulatory compliance 
paradigms and the relationship between regulatory compliance 
and program quality; but this paper deals more specifically with 
program monitoring systems that are being utilized within the 
human services. 


Program Monitoring System’s Key Elements Comparison 


Key Element Uniform Differential Integrated 
1Risk Absolute Relative Relative 
2Rules Equal Not equal Not the focus 
3Quality Standards Not the focus Not the focus Focus 
4Measurement Nominal Nominal Ordinal 
5Approach Everyone gets the | Based on need Open ended 
same 
6Weights None Equal or Relative Balance 
7Philosophy Do no harm Do no harm Do things well 
8Data distribution Linear Non-linear Linear 
9Risk/Performance Risk Risk Performance 
10Scaling 100 or 0 100 or 0 100 -0 
11Function Gatekeeper Gatekeeper Enabler 
12Quality Structural Structural Process 
13Compliance Full Substantial Full/Substantial 
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Risk is defined in a uniform monitoring system with all 
rules at an equal risk level. In differential monitoring, 
risk changes to be more relative in that certain rules are 
more of a concern than others. In an integrated 
monitoring system with the influx of quality elements, 
risk is relative also because of this added dimension. 
Rules are either created equally, which is the case with 
uniform monitoring systems, or they are not equal in 
differential monitoring systems where weights are 
employed to demonstrate the relative risk of specific 
rules. In integrated monitoring systems, rules are 
replaced with standards and specific health and safety 
rules are not the focus. 

Quality standards are the focal point of integrated 
monitoring systems but not so with uniform and 
differential monitoring systems which emphasize health 
and safety rules. 

Measurement at both the uniform and differential 
monitoring systems levels are nominal in which either a 
rule is in or out of compliance. Integrated monitoring 
systems which deal with program quality are generally 
at an ordinal, Likert level of measurement. 

The approach of each of the monitoring systems varies 
from everyone gets-the- same for uniform monitoring 
systems to based-on-need for differential monitoring 
systems, and more open ended for integrated monitoring 
systems where both compliance and quality are equally 
important. 

Weights are not an issue with uniform monitoring 
systems because all rules are dealt with equally and 
therefore are dealt with as strictly violation data with an 
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10. 


11. 


equal weight. With differential monitoring systems that 
is not the case and is the focal point in this approach 
where weights can be either equally applied with a 
Likert Scale with an equal interval or relatively applied 
with the Fibonacci Sequence. Integrated monitoring 
systems have a more balanced approach dependent upon 
the balance of compliance and quality. 

Philosophy for the uniform and differential monitoring 
systems deals more with rules and “do no harm” while 
integrated monitoring systems focus on quality and 
“doing good” or best practices. 

Data distributions are linear when dealing with uniform 
and integrated monitoring systems, but differential 
monitoring systems have clearly demonstrated a non- 
linear data distribution based upon the theory of 
regulatory compliance**. 

Risk/Performance plays out with risk being 
predominant with uniform and differential monitoring 
systems but performance being predominant with 
integrated monitoring systems where quality is central. 
Scaling is at a nominal level in both uniform and 
differential monitoring systems where measurement is 
based upon either being in or out of compliance with 
rules (100 or 0). Integrated monitoring systems are at 
an ordinal level where various levels of quality are being 
assessed (100 —0). 

Function of the approach is either as gatekeeper at both 
the uniform and differential monitoring systems levels 
and as an enabler at the integrated monitoring systems 
where it is more of an open system rather than a closed 
system which is based upon licensing. Open systems 
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are represented by voluntary systems dealing with 
quality standards. 

12. Quality at the uniform and differential monitoring 
systems is more structural than process-oriented, as with 
integrated monitoring systems. In legal terms, it is the 
difference between soft data in the case of process- 
oriented quality as versus hard data in the case of 
structural quality. 

13. Compliance needs to be fully or 100% compliant in 
uniform monitoring systems, which is not the case in 
differential monitoring systems where substantial 
regulatory compliance is sufficient based upon the 
results of the theory of regulatory compliance**. With 
integrated monitoring systems there is more of a 
balancing act between full and substantial compliance 
levels. 


Hopefully, this clarifies how the various program monitoring 
systems used within the human services are similar and 
different. This paper should be read with the other technical 
research papers dealing with regulatory compliance and 
program quality paradigms which enhance upon these above 
stated elements. 


References: 


*Regulatory Compliance Monitoring Paradigms and the Relationship 
of Regulatory Compliance/Licensing with Program Quality: A Policy 
Commentary, Fiene, 2023. Journal of Regulatory Science. 
(https://doi.org/10.21423/JRS-V10A239). 


**Treatise on Regulatory Compliance, Fiene, 2019. Journal of 
Regulatory Science https://doi.org/10.21423/jrs-v07fiene 
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Regulatory Compliance Scale Trials and Tribulations 


The Regulatory Compliance Scale (RCS) was introduced 
several years ago and has been used in a couple of validation 
studies for differential monitoring and regulatory compliance’s 
ceiling effect phenomenon. RCS buckets or thresholds were 
statistically generated based upon these studies, but it is time to 
validate those buckets and thresholds to determine if they are 
really the best model in creating a regulatory compliance scale. 
Since proposing the RCS, there has been a great deal of interest 
from jurisdictions in particular from Asian and African nations. 
Additional statistically based trials were conducted, and this 
brief report is the compilation of those trials over the past year. 


The data used are from several jurisdictions that are part of the 
international database maintained at the Research Institute for 
Key Indicators Data Laboratory at Penn State University 
focusing on program quality scores and rule violation frequency 
data. These data from the respective databases were recoded 
into various thresholds to determine the best model. The 
jurisdictions were all licensing agencies in the US and Canada 
geographically dispersed where both regulatory compliance 
and program quality data was obtained from a sample of early 
care and education programs. 


METHODOLOGY 


The following methodology was used starting with the original 
RCS buckets/thresholds of Full, Substantial, Medium, and Low 
regulatory compliance: 
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Table 1: RCS Models used for analyses 


RCS Models 
Original 1 2 3 4 
Full 100 100 100 100 100 
Scaling | Substantial 99-98 99-97 99-97 99-98 99-98 99-97 
Medium 97-90 96-90 96-93 97-95 97-85 96-85 
Low 89> 89> 92> 94> 84> 


Five alternate models were used to compare the results to the 
original RCS. The numbers indicate the number of violations 
subtract from a perfect score of 100. Full regulatory 
compliance indicates no violations and a score of 100 on the 
scale. The next bucket of 99-98 indicates that there were | or 2 
regulatory compliance violations which resulted in a 99-98 
score on the scale. This logic continues with each of the 
models. 


The scale score was determined in the following manner: Full 
Regulatory Compliance = 7; Substantial Regulatory 
Compliance = 5; Medium Regulatory Compliance = 3; and Low 
Regulatory Compliance =1. This rubric is how the original 
RCS scaling was done on a Likert type scale similar to other 
ECE program quality scales, such as the Environmental Rating 
Scales. 


RESULTS 


The following results are correlations amongst the respective 
RCS Models from Table 1 compared to the respective 
jurisdictions program quality tool (Quality1-3): ERS or CLASS 
Tools. 
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Table 2: RCS Model Results compared to Quality Scales 


RCS results Models Quality1 Quality2 Quality3 
Jurisdiction1 RCSO .26* .39* .39* 
RCS3 21 .32* 43* 
RCS5 .20 .36* 337 
Jurisdiction2 RCSO .76** .46** -—- 
RCS3 12 -.07 --- 
RCS5 18 -.02 --- 
RCSF1 55** .29* --- 
RCSF2 i63** 34 --- 
Jurisdiction3 RCSO 19 18 16 
RCS3 .21 21 15 
RCS5 18 16 .07 
RCSF1 17 alg .10 
RCSF2 18 .18 .19 
Jurisdiction4 RCSO .24* --- -—- 
RCS3 .28* --- --- 
RCS5 .30* = - 
RCSF1 21 --- --- 
RCSF2 .29* --- --- 
Jurisdiction RCSO .06 -.02 .07 
RCS3 .06 -.01 .05 
RCS5 .08 .00 .09 
RCSF1 .00 -.03 .05 
RCSF2 .05 -.03 .05 


*Statistically significant .05 level; 


**Statistically significant .01 level. 


In the above table starting under Jurisdiction2, two new models 
were introduced based upon the Fibonacci Sequence 
(Fibonaccil = RCSF1; Fibonacci2 = RCSF2) and their model 
structure is in the following Table 3. The reason for doing this 
is that the Fibonacci Sequence introduces additional variation 
into the scaling process. 
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Table 3: RCS Fibonacci Models 


RCS Models 
Fibonacci 
Original Fibonacci1 Fibonacci2 
Full 100 100 100 
Scaling Substantial 99-98 40 90 
Medium 97-90 20 20 
Low 89> 13 13 


A second series of analyses were completed in comparing the 
RCS models with program quality (Qualityl) by running 
ANOVAs with the RCS models as the independent variable and 


program quality as the dependent variable (Table 4). 
reason for doing this was the nature of the data distribution in 
which there was a ceiling effect phenomenon identified which 
would have had an impact on the correlations in Table 2 above. 
All results are significant at p < .05 level with the exception of 


Jurisdiction2. 


Table 4: ANOVAs Comparing the RCS Models with Program Quality 


The 


Jurisdictions Model Level 1 Level 3 Level 5 Level 7 
Jurisdiction1 RCSO 2.85 3.34 4.05 3.40 
RCS3 3.24 3.23 4.05 3.40 
RCS5 2.73 3.32 3.77 3.40 
Jurisdiction2 RCSO 4.81 4.31 4.80 4.10 
RCS3 4.59 4.25 4.80 4.10 
RCS5 -- 4.26 4.64 4.10 
Jurisdiction3 RCSO 4.59 4.68 4.86 4.87 
RCS3 4.38 4.67 4.83 4.87 
RCS5 4.38 4.83 4.83 4.87 
Jurisdiction4 RCSO 37.81 37.01 44.28 41.96 
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RCS3 36.57 38.60 44.28 41.96 

RCS5 33.46 36.53 43.10 41.96 

Jurisdictions RCSO 3.93 4.17 4.28 4.07 

RCS3 4.02 4.24 4.28 4.07 

RCS5 3.75 4.13 4.26 4.07 
DISCUSSION 


Based upon the above results, it appears that the original RCS 
model proposed in 2021 is still the best model to be used, 
although the Fibonacci Sequence model is a close second in 
some of the jurisdictions. This model will need further 
exploration in determining its efficacy as a replacement or 
enhancement to the original RCS Model. 


The bottom line is that the original RCS Model is as good as 
any and no other model is consistently better than all the rest. 
The RCS Model does have a slight edge over Regulatory 
Compliance Violation RCV _ frequency counts in some 
jurisdictions but not in others. It is much easier to interpret the 
relationship between quality and the RCS models than it is to 
interpret the results from the quality scores and the RCV data 
distribution. So, the recommendation would be for licensing 
agencies to think in terms of using this new scaling technique 
in one of its model formats in order to determine its efficacy. 
Pairing up RCS and RCV data side by side by licensing 
agencies would be important studies to determine which 
approach is the better approach. 


The below graphic depicts the relationship between the RCS 
Models (0, 3, 5) when compared to the quality scores (1-6) 
clearly showing the ceiling effect and diminishing returns effect 
phenomenon so typical of regulatory compliance data when 
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compared to program quality. These graphs are from the first 
three jurisdictions (1, 2, 3) from the above tables. 


RCS Models Compared to Quality Scores 
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Fiene Methodologies Timeline* 


1970’s Original Conceptual Phase 

1971 Original Concept of Fiene Methodologies. 

1972 Early Childhood Program Quality Indicator Model 
ECPQIM created which is a comprehensive program 
monitoring system measuring regulatory compliance and 


quality initiatives through math modeling via the Fiene 
Methodologies. 


1973 Research methodology studies begun. Master’s Thesis 
1974 ECPQIM presentations done nationally. 


1975 ECPQIM applied to PA ARC child development 
programs, EMIS: Ecological Monitoring Information System 
created (ECPQIMO). 


1976 EMIS utilized throughout PA ARC child development 
programs. 


1977 Child Development Program Evaluation (CDPE) 
monitoring system created and implemented in Pennsylvania. 


1978 Research methodology studies completed. Doctoral 
Dissertation. 


1980’s State Transfer Phase 


1980 Federal Grant to develop Fiene Methodologies to be 
applied at the state level. 


1981 Children’s Services Monitoring Transfer Consortium 
created. 
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1982 Instrument Based Program (IPM) Monitoring created. 


1983 Key Indicator Methodology fully developed. Theory, 
Implementation, Software are all developed and begin transfer 
to state agencies. 


1984 Validation studies completed on key indicators (KIM), 
risk assessment (RAM) and differential monitoring (DM) in 
five states: Pennsylvania, California, West Virginia, Michigan, 
and New York. 


1985 Child Care Quarterly article published introducing IPM, 
KIM, RAM, and DM; and the New England Journal_of 


Human Services article published introducing the Regulatory 
Compliance Theory of Diminishing Returns. 


1986 Expansion of the Fiene Methodologies to other interested 
states and to Canadian Provinces. 


1987 Expansion of the Fiene Methodologies to child welfare 
services through the publications of Instrument Based 


Program Monitoring for Child Welfare, published by the 


University of Southern Maine. 


1990’s National Transfer Phase 


1991 Caring for Our Children created and published. 


1992 NECPA: National Early Childhood Program 
Accreditation adopts the Fiene Methodologies in developing 
their early childhood program accreditation system. 


1993 Stepping Stones for Caring for Our Children created 


utilizing the Risk Assessment Methodology. 
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1995 Solution to the Child Care Trilemma published by Child 
Care Information Exchange utilizing the Key Indicator 
Methodology to solve the trilemma of affordability, 
accessibility, and quality. 


1998 Creation of the Pennsylvania Office of Human Services 
Licensing with a research division devoted to the development 
and implementation of the software development of the Fiene 
Methodologies 


1999 Development of the software for the Fiene 
Methodologies. Pilot tested but never went statewide. 


2000’s_ Professional Development Phase 


2000 Capital Area Early Childhood Research and Training 
Institute (CAECTI created at Pennsylvania State University. 


2001 New coaching intervention implemented and evaluated at 
CAECTI and extended to Pennsylvania’s QRIS KeyStone 
STARS technical assistance program. 


2002 Coaching study published, Assistant Secretary Office of 
Planning and Evaluation’s 13 Indicators of Quality Child Care 
Research __Report published, and Pennsylvania _Early 


Childhood Quality Study published. All three reports support 
the Key Indicator Methodology identification of quality 


indicators. 


2008 New coaching intervention approach implemented in the 
new Human Development and Family Relations program at 
Penn State Harrisburg that emphasized a problem solving 
approach rather than one size fits all. 
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2010’s National Monitoring and QRIS Validation Phase 


2010 Cruise Industry started to use the Key Indicator 
Methodology in the monitoring of their youth programs. 


2012 Conceptual_Framework for QRIS Validation and the 
ECPOIM _ Differential Monitoring Logic Model and 


Algorithm published. 


2013 The state of Washington began to use the term “Fiene Key 
Indicators”. RIKI: Research Institute for Key Indicators created 
and becomes an affiliated data laboratory to the Edna Bennett 
Pierce Prevention Research Center at the Pennsylvania State 
University. RIKI becomes the international repository for the 
Fiene Methodologies. 


2014 Head Start revises their grantee monitoring system 
utilizing the Key Indicator Methodology resulting in the Head 


Start Key Indicator Tool. 


2015 Exclusive partnership with NARA in the future 
development of the Fiene Methodologies. This partnership did 
not apply to software development because at the time NARA 
did not have that capability. Therefore, a batch-oriented 
research file analysis plan was utilized with the hope to develop 
the necessary site-specific online software in the future. Caring 


for Our Children Basics published by Administration for 
Children and Families at the Federal Level. 


2017 CCDBG reauthorization with reference to encouraging 
states to explore differential monitoring and key indicators & 
risk assessment methodologies. It was determined in a Federal 
Report that most states were using a form of differential 
monitoring. 
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2018 The Province of British Columbia, Canada started to use 
the term “Fiene Coefficients” to describe Phi Coefficients 
generated by the Key Indicator Methodology. 


2020’s Software Development Phase 


2020 Final validation studies started with NARA via the 
Province of Saskatchewan to prove the effectiveness and 
efficiency of the Fiene Methodologies. 


2021 It became clear that the software validation would need to 
occur with a software company and not with NARA. 


2022 Development and implementation of the Early Childhood 
Program Quality Indicator Tool for the Province of 
Saskatchewan. First use of the Regulatory Compliance Scale 
utilized in the Saskatchewan Studies. 


2023 Final Saskatchewan validation report completed and 
published. This completes the theory for the Fiene 
Methodologies but the Software still needs to be fully 
developed and validated moving it from a batch-oriented 


research file platform. Regulatory Compliance Monitoring 
Systems etextbook published. 


2024 Exclusive partnership developed with SansWrite/Outlier 
Tech to validate the Fiene Methodologies: Key Indicator, Risk 
Assessment, and Differential Monitoring software. 


2025 Software Validation Studies proposed. 


2026 Proposed launch of the Fiene Methodologies software as 
part of SansWrite. This will totally complete the Fiene 
Methodology validation of both the theory through NARA and 
the software through Outlier Technologies. In moving forward, 
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a joint partnership between the two organizations makes the 
most sense in maintaining the Fiene Methodologies. 


2024 and beyond are proposed that is why they are italized and 
we will see where the journey actually ends. I have made 
suggestions based upon my background as a _ research 
psychologist and regulatory scientist but I have learned over the 
years that other forces can be more influential than science. So 
we will see!? 


*The Fiene Methodologies were started to be referred to in the 
research literature in the 2010’s in various jurisdictions. 
Initially when they were created, the Fiene appellation was not 
used since it was implied via citations since Fiene was the 
author and creator of the methodologies. 


307 


Glossary to the Fiene Methodologies 


Key Indicators. Also known as predictor rules. Caring for 
Our Children Basics is an example of key indicators at the 
national level as well as the Head Start Key Indicator Tool. 
CDPES: Child Development Program Evaluation Scale is an 
example of both licensing and program quality indicators. The 
The Early Care and Education Quality Indicators Scale is an 
example of the program quality indicator tool. Key indicator 
rules predict overall regulatory compliance. In the most recent 
NARA Licensing Study, 10 states use some form of the key 
indicator methodology. In certain jurisdictions (states and 
provinces), key indicators have been referred to as Fiene 
Indicators or the Fiene 13 Indicators as well as the results being 
termed the Fiene Coefficients. |The Assistant Secretary’s 
Office for Planning and Evaluation published a summary of key 
indicator research entitled: The 13 Indicators of Quality Child 
Care: Research Update which highlights the factor analyses of 
key indicator results from several jurisdictions. The research 
brief focuses on health and safety standards. The National 
Center for Health and Safety in Child Care published a Parent’s 
Guide to Choosing Safe and Healthy Child Care which is 
based upon the key indicator methodology. There is also a 
national accreditation program NECPA: National_Early 
Childhood Program Accreditation based upon the key 
indicator methodology utilized by the National Child Care 
Association. 
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From National Association for Regulatory Administration 
NARA’s materials: Key indicator system. A _ licensing 
measurement system utilizing a shortened or abbreviated 
version of a comprehensive checklist measuring compliance 
with rules through a statistical methodology. Only key predictor 
rules are included on an indicator checklist. It is a form of 
differential monitoring or inferential inspections where only a 
portion of the full set of rules is measured. 


Risk Assessment. Also known as risk assessment rules or 
weighted rules. These are rules that place clients at greatest risk 
of morbidity or mortality. They do not predict regulatory 
compliance as do key indicator rules. Risk assessment rules are 
determined through a Likert type measurement methodology. 
Stepping Stones to Caring for Our Children is an example of 
a national risk assessment tool. In the most recent NARA 
Licensing Study, 35 states use some form of the risk assessment 
methodology. 


From NARA’s materials: Risk Assessment/Weighting 
System. A Risk Assessment/Weighting system is a Likert type 
of measurement that utilizes a modified Delphi technique to 
determine the relative risk to individuals if there are violations 
with specific rules. Risk assessment/weighting systems are 
developed by sending a survey to a selected sample of 
persons/stakeholders in order for them to rank the relative risk 
of violation with specific rules. 


Differential Monitoring. Also known as_ abbreviated 
inspections or inferential inspections. Differential monitoring 
includes both key indicator and risk assessment methodologies 
as it relates to this approach. But differential monitoring can be 
either of these two methodologies or it can focus on the number 
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of inspections that are done, for example, there may be fewer 
inspections if a provider is eligible for a differential monitoring 
approach. Differential monitoring when utilized within a 
validation framework approach may be referred to as the 
Differential__Monitoring Logic _Model__and__Algorithm 
(DMLMA) which incorporates a mathematical modeling 
approach to dealing with the four stages of validation. 


From NARA’s materials: Differential 
monitoring/inferential inspections. An abbreviated 
inspection utilizing a select set of rules to be reviewed. A key 
indicator system or a risk assessment system are two examples 
of differential monitoring approaches. The use of differential 
monitoring/inferential inspections by licensing agencies was 
developed as a time saving technique and a technique to focus 
regulatory efforts on facilities that required additional 
inspections or technical assistance. 


Quality Indicators. These are key indicators generated 
utilizing the key indicator methodology but with program 
quality standards rather than licensing rules and regulations. 
Examples include the Child Development Program Evaluation 


Scale and the The Early Care and Education Quality 
Indicators Scale. Both tools go beyond the standard key 


indicator methodology for rules and regulations. 


Early Childhood Program Quality Indicator Model 
(ECPQIM). The ECPQIM is a comprehensive program 
monitoring model that measures regulatory compliance and 
quality initiatives: professional development systems and 
Quality Rating & Improvement Systems (QRIS). It utilizes 
comprehensive tools taken from Caring for Our Children, 
State Regulations, and the Environmental Rating Scales as 
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well as the predominant differential monitoring approaches of 
key indicator and risk assessment methodologies. The 
DMLMA mentioned above is a key component of the ECPQIM 
in that it provides the mathematical backbone of the model as 
well as providing the algorithms for the Validation Conceptual 
Framework as outlined by Zellman and Fiene (2012). 


From NARA’s materials: Instrument based program 
monitoring. A movement within licensing and regulatory 
administration from qualitative measurement to a_ very 
quantitative form of measurement that includes the use of 
checklists. This move to more quantitative has been encouraged 
as more and more states develop electronic data systems. 


From NARA’s materials: Checklist. A checklist is a simple 
measurement tool that measures compliance with state rules in 
a yes/no nominal format. Either the facility is in compliance 
with rules or not in compliance. Generally, there is no partial 
compliance with checklists. Having regulatory compliance data 
being at a nominal measurement level creates limitations 
statistically in the types of tests that can be completed. 


From NARA’s materials: Rating Scale. A rating scale is a 
more complex measurement tool in which a Likert type of 
rating is employed going from more to less or high to low. A 
rating scale is always used in the development of weighting/risk 
assessment systems. It is not used in measuring compliance 
with rules or at least it hasn’t been used in the past. 


From NARA’s materials: Outcome based Systems. Outcome 
based systems are measurement systems based upon outcomes, 
not processes. A facility would be assessed by the outcomes it 
produced with individuals. For example, the number of 
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consumers (children or adults) developing normally, free from 
abuse, not in placement, involved actively in the community, 
properly immunized, free from injuries, etc. are outcome-based 
measures. 


I hope you enjoyed your journey in learning about regulatory 
compliance and came across some new ideas and concepts. For 
the interested reader, I encourage you to check out the NARA 
and RIKI websites which contain a great deal more resources to 
support this book. [have mentioned both repeatedly throughout 
the text providing their websites. 


I plan on continuing to research enhancements to the 
methodologies which now bear my name to determine the most 
effective and efficient approaches as we enter Version 2.0. 


Thank you, 
Rick 
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I just couldn’t resist sharing with you a potential extension 
that I hope becomes part of Version 2.0. The theory of 
regulatory compliance has been critical in the development of 
differential monitoring. Here it is extended into another arena 
in the development of the regulatory compliance scale. 


The Theory of Regulatory Compliance, Regulatory 
Compliance Scale, and Differential Monitoring 


The theory of regulatory compliance has been proven in 
multiple studies over the past four decades and has been utilized 
extensively in the creation of differential monitoring and its 
spin off methodologies of risk assessment and key indicators. 
In fact, differential monitoring would not have been possible 
without the theory of regulatory compliance because the 
paradigm which it replaced, one of one-size-fits-all monitoring 
or uniform monitoring would have predominated. However, 
with the theory of regulatory compliance which introduced the 
importance of substantial regulatory compliance and the search 
for the right rules/regulations that made a difference in client’s 
lives, rather than emphasizing more or less regulations or rules. 


The theory of regulatory compliance has another application 
when it comes to regulatory compliance measurement in 
helping to move the licensing field from a nominal based 
measurement strategy to one of ordinal based measurement. 
The new measurement strategy is the Regulatory Compliance 
Scale (RCS) and it is depicted in the following table. 
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RCS Compliance Risk Model Model 
Scale Level Level Violations Weights 
7=A Full None 0 0 
5=B Substantial Low 1-3 1-3 
3=C Medium Medium 4-9 4-6 
1=D Low High 10+ 7+ 


The above table needs some explanation. The first column is 
the proposed ordinal scale similar to other scales utilized in the 
program quality measurement research literature on a 1 — 7 
Likert Scale where 7 = Full Regulatory Compliance, 5 = 
Substantial Regulatory Compliance, 3 = Medium Regulatory 
Compliance, and 1 = Low Regulatory Compliance. It could 
also be thought of as an Alpha Scale of A—D as well. The next 
column has the compliance levels that run from full 100% 
regulatory compliance to low regulatory compliance. The third 
column depicts the risk level from none to high which 
corresponds with the compliance levels. The next two columns 
depict two models, one unweighted and one in which the rules 
are weighted with corresponding weights. These models are 
based upon the two prevailing approaches to rank ordering rules 
or regulations in the research literature. 


The following figures will depict how the scale was conceived 
based upon empirical evidence in the various studies supporting 
the theory of regulatory compliance. The first figure shows the 
actual individual violation data (LSTOTAL) of the programs 
compared to their corresponding ECERS (ECERSPS) scores. 
There is not a significant relationship between the two as 
depicted in the graphic. 
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Scatterplot ECERSPS vs. LSTOTAL 
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The following figure below depicts what occurs when the 
individual violation data are grouped according to the theory of 
regulatory compliance in which a substantial compliance 
category is introduced and the data are moved from a nominally 
based metric to an ordinally based metric of full, substantial, 
medium, and low regulatory compliance categories (RCS). 


This grouping more clearly reflects the theory of regulatory 
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compliance. It also clearly demonstrates the ceiling effect 
which is an outcome of the theory of regulatory compliance in 
which substantial and full regulatory compliance levels are 
basically equivalent when quality is taken into account. Or at 
the extreme level which is depicted here where full regulatory 
compliance quality scores are actually lower than the 
substantial regulatory compliance quality scores. A footnote 
about the figures and the scaling: the scales for the first figure 
are on a lower to higher progression but the higher LSTOTAL 
represents higher non-compliance where the second figure is 
also based upon lower to higher scaling but the higher scores 
represent increased quality (ECERSPS) and _ increased 
regulatory compliance (RCS). So, in reading the change from 
left to right, the two figures are reversed images of each other. 
This is just a quirk of the scaling and not a mistake in the 
plotting of data. 


The RCS has been pilot tested in both the non-weighted and 
weighted models and based upon these studies it appears to be 
more effective in distinguishing quality amongst the various 
categories rather than utilizing violation count data. This would 
be a significant improvement when it comes to licensing 
measurement. Of course, additional replication studies need to 
be completed before it would be recommended as a new Scale 
to be used for making licensing decisions. 
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Scatterplot ECERSPS vs. RCS 
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The above figure is dramatically different than the prevailing 
paradigm which predicts a linear relationship between 
regulatory compliance and quality which is the paradigm of a 
uniform monitoring approach. The above results clearly 
indicate a reconsideration with the introduction of substantial 
regulatory compliance as an important contributor to overall 
quality if not the most important contributor to quality. As 
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stated above, these findings have been replicated in several 
studies conducted over the past several decades. 


This would be a major paradigm shift in moving from 
individual violation data counts to an ordinal scale metric but it 
does warrant additional research. The problem with individual 
violation data is that it doesn’t take into account the relative risk 
of the individual rule which could place clients at increased risk 
of morbidity or mortality. Risk assessment has worked really 
well when coupled with key indicators in the differential 
monitoring approach and it appears to be an asset in the 
development of a Regulatory Compliance Scale (RCS). 


And here is another thought for you to think about when it 
comes to risk assessment for Version 2.0: 


Could the Fibonacci Sequence be Superior to the Equal 
Interval Weighting for Risk Assessment? 


Risk assessment (RA) has generally used a 3 x 3 matrix similar 
to the one depicted in Table 1 below where an equal interval 
weighting was utilized in describing the nine cells that 
constituted the 3 x 3 matrix. The matrix considers the risk 
levels and compares that to the probability of the actual risk 
occurring, so there were nine possibilities (1 — 9). 
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Table 1: RA Likert Absolute Equal Interval Weighting 


Uniform Risk Risk Levels 
High Medium Low 
High 9 6 
Probability 
of Medium 8 B) 2 
Occurring Low 7 4 1 


More recently, a proposed change has been suggested to utilize 
the Fibonacci Sequence in place of the equal interval weighting. 
There is a great deal of merit in considering this because the 
Fibonacci Sequence has an interesting effect in introducing a 
differential risk function rather than a more uniform risk as is 
the case with equal interval weighting as depicted in table 1. In 
table 2, it is clear the Fibonacci Sequence has a tremendous 
impact on the increasing value of the various risk/probability 
cells within the 3 x 3 matrix. It mirrors the increasing risk, 
which considers the risk already present in the previous cell and 
then increases it by the next numeric increase. This increase 
changes markedly as the risk/probability goes up by a factor of 
over 10 at the highest risk level. The nine cells range from | — 
100 rather than | - 9. 


Table 2: RA Relative Weighting: The Fibonacci Sequence 


Differential Risk Risk Levels 
High Medium Low 
High 100 13 3 
Probability 
of Medium 40 8 2 
Occurring Low 20 5 1 
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The above sequence is not an exact Fibonacci Sequence and 
modifies the cell results at the high-risk levels to accentuate this 
level. A potential proposal is depicted in table 3 in how this 
could play out with the weighting of rules/regulations related to 
the health and safety of clients and their relative risk of 
mortality and/or morbidity because of non-compliance with 
such rules/regulations either directly or indirectly. 


Table 3: RA Relative Weighting: The Fibonacci Sequence Proposal 


Differential Risk Risk Levels 
Direct (Causality) Indirect 
(Correlation) 
Mortality Morbidity Mortality/Morbidity 
High Medium Low 
High 100 13 3 
Probability 
of Medium 40 8 2 
Occurring Low 0 5 1 


This proposal needs to be tested and compared to the more 
prevalent equal interval weighting approach to see if it is a 
better predictor in identifying the risk level of rules and 
regulations when it comes to health and safety. 


Some final thoughts about the enhancements for DMS 2.0 that 


I think are important take aways from what we are learning in 
moving from DMS 1.0 to DMS 2.0. 


Relative weighting is a group exercise, not an individual one 
which is the case with absolute equal interval weighting as 
represented in DMS 1.0. This aligns with the latest research in 
the psychological literature about how much more effective a 
group-oriented process is when _ statistically comparing 
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consensus on a new concept, such as_ weighting 
rules/regulations. 


The theory of regulatory compliance highlighted the 
importance of substantial regulatory compliance as being as 
effective as full regulatory compliance and in some cases more 
effective when it comes to identifying quality programs. This 
result has led to a major revision in the development of a scoring 
system for regulatory compliance, the Regulatory Compliance 
Scale which reflects the theory. 


There appears to be an important relationship between relative 
weighting and substantial compliance. In DMS 1.0 it was 
necessary to make adjustments to the key indicator 
methodology when substantial compliance was introduced into 
the high regulatory compliance group. It increased the number 
of false negatives. In an unweighted or an absolute equal 
interval weighting approaches this makes sense because higher 
risk rules may slip into the equation. The work around was two- 
fold: 1) Only include full regulatory compliance programs in 
the high group or 2) Increase the false negative cell to the point 
that it was relatively impossible not to include the results from 
this cell because of regulatory compliance non-compliance. 


However, with relative weighting, this is not the case. The 
relative weighting because of the increased variance in the data 
and pushing the regulatory compliance scale that accounts very 
sensitively to high-risk rules. This is not the case with 
unweighted or absolute equal interval weighting approaches. 


A real advantage in using the regulatory compliance scale is that 
it moves licensing measurement from a nominal to an ordinal 
level of measurement based upon the theory of regulatory 
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compliance where there are four levels of regulatory 
compliance: full, substantial, medium, and low. These levels 
fit perfectly with the theory, especially with the introduction of 
the substantial regulatory compliance level as its own 
individual level. 


DMS 1.0 clearly demonstrated the importance of the 
Regulatory Compliance Scale. The limitations of just using 
violation counts were demonstrated in DMS 1.0 as well and 
should be replaced now with relative weighting within the Key 
Indicator methodology. 


Risk rules are generally always in compliance. Because of their 
greater chance to be associated with mortality or morbidity, 
these risk rules are usually not out of compliance. Key Indicator 
rules are generally somewhere in the mid-range when it comes 
to regulatory compliance. Not particularly high non- 
compliance but also no high compliance either; they are 
generally somewhere in between and to be key indicator rules 
they are generally in compliance with the high regulatory 
compliant group and out of compliance with the low regulatory 
compliant group. So, these are the good discriminator rules in 
determining overall comprehensive regulatory compliance. In 
conclusion, full regulatory compliance equates to a healthy and 
safe environment, but it does not necessarily mean it is of the 
highest quality. Within a regulatory compliance schema, 
substantial compliance appears more related to program 
quality. Risk assessment rules are always in regulatory 
compliance in either one of these scenarios. 


The Risk Rule Matrix generally has risk values that fall within 
the mid risk level with fewer instances in the high and low risk 
levels. In general rule of thumb is 25% low risk, 50% mid level 
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risk or higher, and 25% high risk or lower; however, there are 
jurisdictions which weight their rules and regulations a bit 
higher and you will see a higher percentage in the higher risk 
level. In DMS 2.0, these results will need to be monitored 
closely to see how this plays out with relative weighting in 
comparison to equal interval weighting. 


And lastly, the regulatory compliance scale which moves 
licensing measurement from a nominal level to an ordinal level 
provides an enhanced measurement strategy more similar to the 
program quality scales such as the Environmental Rating Scales 
and the CLASS. It is a major improvement over just using 
violation count data but still needs additional empirical 
evidence to continue to prove its efficiency and effectiveness, 
especially now with the introduction of relative weighting of 
rules and regulations. 
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Summary generated by CoPilot: 


Purpose: The eHandBook provides an introduction to 
regulatory compliance, licensing measurement, and 
program monitoring systems for researchers, 
administrators, and policymakers. 


Content Overview: It covers topics such as conceptual 
frameworks, instrument design, program quality, 
coordinated monitoring, and future directions. 


Historical Context: The preface discusses the evolution 
of regulatory science and the development of licensing 
measurement methodologies over the past 50 years. 


Target Audience: The book is intended for those 
involved in regulatory science, particularly in early 
care and education, and is part of the NARA Licensing 
Curriculum. 
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About the Author: 


After a long career in governmental service and academia, 
mostly in Pennsylvania; and consulting, nationally and 
internationally, Dr Rick Fiene continues to write and research 
about regulatory science topics (such as measurement, 
instrument development, math & statistical modeling, 
differential monitoring, risk assessment, key performance 
indicators) as they relate to early care and education, the 
human services, and has been delving into other social 
sciences as well. 


https://www. prevention. psu. edu/people/rick-fiene 


Contact Dr Fiene at his Research Institute or through the Penn 
State Prevention Research Center: 


rfiene @rikinstitute.com 


or through the National Association for Regulatory 
Administration: 


rfiene @naralicensing.org 
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